Fusing depth and pressure imaging to provide object identification for multi-touch surfaces

ABSTRACT

An apparatus for inputting information into a computer includes a 3d sensor that senses 3d information and produces a 3d output The apparatus includes a 2d sensor that senses 2d information and produces a 2d output The apparatus includes a processing unit which receives the 2d and 3d output and produces a combined output that is a function of the 2d and 3d output. A method for inputting information into a computer. The method includes the steps of producing a 3d output with a 3d sensor that senses 3d information. There is the step of producing a 2d output with a 2d sensor that senses 2d information. There is the step of receiving the 2d and 3d output at a processing unit. There is the step of producing a combined output with the processing unit that is a function of the 2d and 3d output.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. provisional patentapplication 61/404,897 filed Oct. 12, 2010; and from U.S. provisionalpatent application 61/462,789 filed Feb. 8, 2011; and from U.S.provisional patent application 61/572,642 filed Jul. 19, 2011; and fromU.S. provisional patent application 61/572,938 filed Jul. 25, 2011, allof which are incorporated by reference herein.

FIELD OF THE INVENTION

The present invention is related to a sensor which reconstructs acontinuous position of force on a surface from interpolation based ondata signals received from a grid of wires. (As used herein, referencesto the “present invention” or “invention” relate to exemplaryembodiments and not necessarily to every embodiment encompassed by theappended claims.) More specifically, the present invention is related toa sensor which reconstructs a continuous position of force on a surfacefrom interpolation based on data signals received from a grid of wireswhere the sensor includes a plurality of plates and a set ofprotrusions.

The present invention relates to receiving at a computer 2d and 3doutput from a 2d sensor and a 3d sensor and producing with the computera combined output that is a function of the 2d and 3d output. Morespecifically, the present invention relates to receiving at a computer2d and 3d output from a 2d sensor and a 3d sensor and producing with thecomputer a combined output that is a function of the 2d and 3d output,where the 2d sensor senses imposed force on its surface and the 3dsensor is a camera.

BACKGROUND OF THE INVENTION

This section is intended to introduce the reader to various aspects ofthe art that may be related to various aspects of the present invention.The following discussion is intended to provide information tofacilitate a better understanding of the present invention. Accordingly,it should be understood that statements in the following discussion areto be read in this light, and not as admissions of prior art.

In prior art, Rosenberg et al teach how to capture a time-varying twodimensional array of pressure upon a surface in a way that properlyinterpolates sensed pressure at points between individual sensingelements. This is an improvement over previous methods, such as that ofTekScan, which do not interpolate between sensing elements, andtherefore must use a very finely spaced two dimensional sensing elementarray to approximate capture of the continuous pressure image.

Moreover, Gesture sensing based only on range imaging cameras can bevery powerful, since it can track entire hand or foot movements,maintain consistent identity over time of each hand of each user, and insome cases provide unambiguous finger and toe identity (depending ondistance of camera to surface and hand or foot position). This stands inmarked contrast to purely surface-based Touch Devices, such as thosebased on variable resistance or capacitance, which provide little or noinformation about finger and hand position or toe and foot position inthe space above the surface. Yet range imaging camera suffers fromseveral deficiencies:

(1) Frame rate (30 fps for the Kinect) is too slow to properly samplethe movement of a finger pressing down and releasing a key. By way ofcomparison, the standard sampling rate for USB keyboards is 125 Hz (morethan four times video rate). This higher sampling rate is needed forunambiguous detection and disambiguation of multiple overlapping typedkeystrokes.

(2) It is impossible to determine from a range image alone how muchpressure is being applied to a surface, thereby rendering range imagingcameras inadequate for subtle movement of virtual objects on a display,rapid and accurate control of 3D computer game characters, musicalinstrument emulation, simulated surgery, simulated painting/sculpting,gait monitoring, dance, monitoring stance for purposes of physicaltherapy, and other applications that benefit from a significant measureof isometric control.

It is therefore also impossible to determine from a 3D image gesturesbased on movements and variations in pressure on the underside offingers or hands or feet or toes. For example, if a user shifts weightbetween different fingers, or between fingers and different parts of thepalm, or between the foot heel, metatarsal or toes, these changes willbe undetectable to a range imaging camera.

The decade of 2001-2011 has seen the gradual development of LCD displaysthat contain an optically sensitive element in each pixel (variouslydeveloped by Sharp, Toshiba and Matsushita). This approach enables thesensing of both touch and hovering. However, the optically sensitivepixel approach suffers from a number of deficiencies as compared to thepresent touch-range fusion apparatus approach: (1) The cost per unitarea is intrinsically far higher than the cost per unit area of theapproach here; (2) Such sensors cannot be seamlessly tiled toarbitrarily large form factors; (3) variations in the pressure of adetected touch 111 can be determined only with very low fidelity (viachanges in fingertip contact shape); (4) hand shape can only be detectedwithin a relatively small distance above the display. This makes itimpossible to maintain a persistent model of hand and finger identity orto recognize many hand gestures. In addition, it is not practical to usesuch technologies for foot sensing, since the added cost to manufacturesuch sensors so that they possess sufficient physical robustness towithstand the weight of a human body would add prohibitively to theircost.

BRIEF SUMMARY OF THE INVENTION

One key innovation of the current invention is that, unlike Rosenberg etal., this method is able to capture a time-varying two dimensional arrayof pressure upon a surface of arbitrarily large size. Therefore, unlikethe method of Rosenberg et al., the current invention can be used forseamless time-varying pressure capture over entire extended surfaces,such as walls, floors, tables, desks or roadways.

The key innovative techniques of the current invention which enable thiscapability are (1) the organization of the sensing element array intophysically distinct tiles, and (2) a method of interpolation betweensensing elements that can operate across tile boundaries.

Also, because the current invention is based on a strategy of seamlesstiling, it is able to make use of an optimization whereby the resolutionof the sub-array formed by each physical tile is chosen so as to makeoptimal use of a microcontroller that controls the data capture fromthat tile. This permits a uniquely economical implementation to beeffected, whereby control of a tile requires only a single commerciallyavailable microcontroller, without requiring the use of any additionaltransistors or other switchable electronic components.

In addition, a Touch-Range fusion apparatus and software abstractionlayer are described that reliably combine the Pressure Imaging Apparatusor other Touch Device data with the data from one or more range imagingcameras in order to create a high quality representation of hand andfinger action for one or more users, as well as foot and toe action ofone or more users, as well as identify and track pens and other objectson or above a Touch Device. It is believed there is currently notechnology available at the commodity level that provide high qualityinput, over a large-scale surface, of finger-identification, pressure,and hand gesture or foot gesture, with simultaneous support ofidentifiable multiple users. This invention will lead to products thatwill fill that gap.

The present invention pertains to an apparatus for sensing. Theapparatus comprises a computer. The apparatus comprises two or moreindividual sensing tiles in communication with the computer that form asensor surface that detects force applied to the surface and provides asignal corresponding to the force to the computer which produces fromthe signal a time varying continuous image of force applied to thesurface, where the surface is contiguous, detected force can be sensedin a manner that is geometrically continuous and seamless on a surface.

The present invention pertains to a sensor. The sensor comprises a gridof wires that define intersections and areas of space between the wires.The sensor comprises a set of protrusions that are in contact with aplurality of intersections of the grid of wires, and a mechanical layerthat is disposed atop the set of protrusions, so that force imparted tothe top of that mechanical layer is transmitted through the protrusions,and thence to the protrusions. The sensor comprises a computer incommunication with the grid which causes prompting signals to be sent tothe grid and reconstructs a continuous position of force on the surfacefrom interpolation based on data signals received from the grid.

The present invention pertains to a sensor. The sensor comprises acomputer having N dual analog/digital I/O pins and M digital I/O pinsfor data, where M and N are positive integers greater than three. Thesensor comprises a pressure sensing array having N rows and M columns,with the N I/O pins in communication with the N rows and up to M columnsin communication with the M I/O pins without using any transistors orother switchable electronic components outside of the computer.

The present invention pertains to a method for determining locations oftiles of a sensor. The method comprises the steps of sending a querysignal from a computer to at least a plurality of the tiles incommunication with the computer asking each of the plurality of tiles toidentify at least one adjacent tile with which the tile is in electricalcommunication. There is the step of receiving by the computer responsesto the query from the plurality of tile. There is the step of formingwith the computer from the responses a geometric map of the tiles'locations relative to each other.

The present invention pertains to a method for sensing. The methodcomprises the steps of detecting a force applied to a sensor surfaceformed of two or more individual sensing tiles from an object movingacross the surface where the surface is contiguous, detected force canbe sensed in a manner that is geometrically continuous and seamless on asurface. There is the step of providing a signal corresponding to theforce to a computer from the tiles in communication with the computer.There is the step of producing with the computer from the signal a timevarying continuous image of force applied to the surface.

The present invention pertains to a method for sensing. The methodcomprises the steps of imparting a force to a top of a mechanical layerthat is transmitted through to intersections defined by a grid of wireshaving areas of space between the wires. There is the step of causingprompting signals with a computer in communication with the grid to besent to the grid. There is the step of reconstructing with the computera continuous position of the force on the surface from interpolationbased on data signals received from the grid.

The present invention pertains to a sensor. The sensor comprises a gridof wires that define intersections and areas of space between the wires.The sensor comprises a set of protrusions that engage with a pluralityof intersections of the grid of wires, and an outer surface layer havingan inner face that is in juxtaposition with the set of protrusions andan outer face, so that force imparted to the outer face of the outersurface layer is transmitted through the inner face of the outer surfacelayer to the protrusions and the plurality of intersections. The sensorcomprises a computer in communication with the grid which causesprompting signals to be sent to the grid and reconstructs an antialiasedimage of force upon the outer face of the outer surface layer frominterpolation based on data signals received from the grid.

The present invention pertains to a method for sensing. The methodcomprises the steps of imparting a force to an outer face of an outersurface layer that is transmitted through an inner face of the outersurface layer to a set of protrusions and a plurality of intersectionsdefined by a grid of wires having areas of space between the wires.There is the step of causing prompting signals with a computer incommunication with the grid to be sent to the grid. There is the step ofreconstructing with the computer an antialiased image of the force onthe outer face of the outer surface from interpolation based on datasignals received from the grid.

The present invention pertains to a sensor. The sensor comprises a gridof wires that define intersections and areas of space between the wires.The sensor comprises a set of protrusions that are in contact with aplurality of intersections of the grid of wires, and an outer surfacelayer having an inner face that is disposed in contact with the grid ofwires and an outer face, so that force imparted onto the outer face ofthe outer surface layer is transmitted through the inner face of theouter surface layer to the protrusions, and thence to the intersectionsof the grid wires which are thereby compressed between the outer surfacelayer and protrusions; and that the protrusions thereby focus theimparted force directly onto the intersections. The sensor comprises acomputer in communication with the grid which causes prompting signalsto be sent to the grid and reconstructs an antialiased image of forceupon the outer face of outer surface layer from interpolation based ondata signals received from the grid.

The present invention pertains to a sensor. The sensor comprises a gridof wires that define intersections and areas of space between the wires.The sensor comprises a set of protrusions that are in contact with aplurality of intersections of the grid of wires, and a mechanical layerhaving a plurality of plates that is disposed atop the grid of wires, sothat force imparted to the top of the mechanical layer is transmittedthrough the intersections, and thence to the. The sensor comprises acomputer in communication with the grid which causes prompting signalsto be sent to the grid and reconstructs a continuous position of forceon the surface from interpolation based on data signals received fromthe grid.

The present invention pertains to a sensor. The sensor comprises a gridof wires that define intersections and areas of space between the wires.The sensor comprises a set of protrusions that are in contact with aplurality of intersections of the grid of wires. The sensor comprises aplate layer having a plurality of plates that is disposed atop the gridof wires. The sensor comprises a flexible touch layer disposed on theplate layer, wherein force imparted to the touch layer is transmittedthrough the plate layer and at least one protrusion to theintersections. The sensor comprises a computer in communication with thegrid which causes prompting signals to be sent to the grid andreconstructs a continuous position of force on the surface frominterpolation based on data signals received from the grid.

The present invention pertains to a sensor. The sensor comprises a gridof wires that define intersections and areas of space between the wires.The sensor comprises a set of protrusions that are in contact with aplurality of intersections of the grid of wires. The sensor comprises aplate layer having a plurality of plates that is disposed atop the gridof wires. The sensor comprises a flexible touch layer disposed on theplate layer, wherein force imparted to the touch layer is transmittedthrough the plate layer to the intersections layer, and thence to theprotrusions. The sensor comprises a computer in communication with thegrid which causes prompting signals to be sent to the grid andreconstructs a continuous position of force on the surface frominterpolation based on data signals received from the grid.

The present invention pertains to a sensor. The sensor comprises a setof plates that are in contact from the bottom at their corners with aset of protrusions that are in contact from above with a plurality ofintersections, each having a sensing element, of the grid of wires, anda thin top surface layer that is disposed atop the grid of plates, sothat force imparted from above onto the top surface layer is transmittedto the plates and thence to the protrusions, and thence to theintersections of the grid wires which are thereby compressed between thebase and protrusions; and that the protrusions above thereby focus theimparted force directly onto the sensor intersections. The sensorcomprises a computer in communication with the sensor grid which causesprompting signals to be sent to the grid and reconstructs a continuousposition of force on the surface from interpolation based on datasignals received from the grid.

The present invention pertains to a method for sensing. The methodcomprises the steps of imparting force from above onto a top surfacelayer that is transmitted to a set of plates and thence to a set ofprotrusions, and thence to a plurality intersections of a grid of wireswhich are thereby compressed between the base and protrusions, where theset of plates are in contact from their bottom at their corners with theset of protrusions that are in contact from above with the plurality ofintersections of the grid of wires disposed on the base; and that theprotrusions above thereby focus the imparted force directly onto theintersections. There is the step of causing prompting signals by acomputer in communication with the grid to be sent to the grid. There isthe step of reconstructing with the computer a continuous position offorce on the surface from interpolation based on data signals receivedfrom the grid.

The present invention pertains to a sensor. The sensor comprises a setof protrusions that are in contact from the bottom with a plurality ofintersections of the grid of wires, and a set of plates that are incontact from the top with a plurality of intersections of the grid ofwires, and a thin top surface layer that is disposed atop the set ofplates, so that force imparted from above onto the top surface layer istransmitted to the plates, and thence to the intersections of the gridwires, and thence the protrusions, which are thereby compressed betweenthe plates and protrusions; and that the protrusions underneath therebyfocus the imparted force directly onto the sensor intersections. Thesensor comprises a computer in communication with the sensor grid whichcauses prompting signals to be sent to the grid and reconstructs acontinuous position of force on the surface from interpolation based ondata signals received from the grid.

The present invention pertains to an apparatus for inputting informationinto a computer. The apparatus comprises a 3d sensor that senses 3dinformation and produces a 3d output. The apparatus comprises a 2dsensor that senses 2d information and produces a 2d output. Theapparatus comprises a processing unit which receives the 2d and 3doutput and produces a combined output that is a function of the 2d and3d output.

The present invention pertains to a method for inputting informationinto a computer. The method comprises the steps of producing a 3d outputwith a 3d sensor that senses 3d information. There is the step ofproducing a 2d output with a 2d sensor that senses 2d information. Thereis the step of receiving the 2d and 3d output at a processing unit.There is the step of producing a combined output with the processingunit that is a function of the 2d and 3d output.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

In the accompanying drawings, the preferred embodiment of the inventionand preferred methods of practicing the invention are illustrated inwhich:

FIG. 1 shows the active sensing array.

FIG. 2 shows the alignment of two Sensor Surfaces.

FIG. 3 shows schematic of Sensor Surface.

FIG. 4 shows the layers of a Sensor Surface.

FIG. 5 shows schematic of Conductor Trace Lines.

FIG. 6 shows schematic pattern of FSR placement.

FIG. 7 shows schematic of Conductor Trace Lines Test Pattern.

FIG. 8 shows schematic pattern of FSR placement Test Pattern.

FIG. 9A shows a sensor surface with Conductor and FSR Test Patterns.

FIG. 9B shows an active sensing array with Conductor and FSR TestPatterns.

FIG. 10 shows the exploded schematic makeup of a single Sensing element.

FIG. 11 shows the active area of a sensing element.

FIG. 12 shows, at a single sensing element, the layers of elements in anembodiment where the protrusions are integrated onto to the outersurface of the Active Sensing Array.

FIG. 13 shows force imparted upon touch layer in an embodiment where theprotrusions are integrated onto to the outer surface of the ActiveSensing Array.

FIG. 14 shows force imparted upon touch layer between two adjacent tilesin an embodiment where the protrusions are integrated onto to the outersurface of the Active Sensing Array.

FIG. 15 shows, at a single sense, the layers of elements in anembodiment where the protrusions are integrated onto the inner surfaceof the Semi-Rigid Touch Layer.

FIG. 16 shows a view from the body of an embodiment of the semi-rigidtouch layer where the protrusions are integrated into the semi-rigidtouch layer.

FIG. 17 shows layers of elements in an embodiment where the protrusionsare integrated onto the inner surface of the Semi-Rigid Touch Layer at asingle sensing element.

FIG. 18 shows a profile view of the redistributing of pressure betweensensing elements that belong to different physical tiles and alsoshowing the active sensing array wrapped under the tile.

FIG. 19 shows exploded view of tile and the appropriate alignment ofprotrusions and sensing elements for an integrated protrusion and baselayer.

FIG. 20 shows layer of elements in an embodiment with a integratedprotrusion and base layer.

FIG. 21 shows an embodiment where the proposed semi-rigid touch layer isunacceptably too rigid.

FIG. 22 shows an embodiment where the semi-rigid touch layer isacceptably semi-rigid.

FIG. 23 shows an embodiment where the proposed semi-rigid touch layer isunacceptably not rigid enough.

FIG. 24 shows distribution of force imparted upon a semi-rigid touchlayer in an integrated protrusion and base layer embodiment.

FIG. 25 shows a region where force would be distributed to fourprotrusions on the same pressure tile.

FIG. 26 shows a region where force would be distributed to twoprotrusions on each of two adjacent pressure tiles.

FIG. 27 shows a region where force would be distributed to oneprotrusion on each of four adjacent pressure tiles.

FIG. 28 shows tall/narrow protrusions.

FIG. 29 shows hemispherical protrusions.

FIG. 30 shows rounded protrusions wider at the base than the height.

FIG. 31 shows rounded protrusions with base very large relative to itsheight.

FIG. 32 is a side view showing the active sensing array folded under theIntegrated Protrusion and Base Layer embodiment.

FIG. 33 shows the side view showing the active sensing array foldedunder the Integrated Protrusion and Base Layer embodiment.

FIG. 34 shows the bottom view showing the active sensing array foldedunder the Integrated Protrusion and Base Layer, having a cavity for thePCB embodiment.

FIG. 35 shows the use of the single tile sensing apparatus.

FIG. 36 shows the use of the grid of tiles sensing apparatus.

FIG. 37 shows the schematic of a data bus of a grid of tiles using I2 C.

FIG. 38 shows grid of tiles and their electronic connectors.

FIG. 39 shows a multiplicity of zones of grids of tiles.

FIG. 40 shows schematic of tiles with N/S/E/W detection lines.

FIG. 41 shows exploded inter tile alignment connectors.

FIG. 42A shows side view of alignment of inter-tile alignmentconnectors.

FIG. 42B shows side view of inter-tile alignment connectors in position.

FIG. 43 shows a disconnected grid of tiles.

FIG. 44 shows cables/wires to/from Microprocessor.

FIG. 45 shows adjacent tiles preserving inter-sensing element distance.

FIG. 46 shows a block diagram of the electronics for a tile functioningas both the Host communication Tile and as a Master Tile.

FIG. 47 shows a block diagram for a slave tile.

FIG. 48 shows labeled positions for use in compensation function.

FIG. 49 shows a graph of a compensation function.

FIG. 50 shows multiple tiles with common touch layer.

FIG. 51 showing applied force applied to sensing elements on differenttiles in the integrated protrusion and base layer embodiment.

FIG. 52 shows an exploded view of a Tile for the Integrated Plate andProtrusion Matrix Component embodiment.

FIG. 53 shows a profile view of a Tile for the Integrated Plate andProtrusion Matrix Component embodiment.

FIG. 54 shows an exploded view of a Tile for the Distinct Plate andProtrusion Matrix Components embodiment.

FIG. 55 shows a profile view of a Tile for the Distinct Plate andProtrusion Matrix Components embodiment.

FIG. 56 shows an embodiment where the protrusions are affixed to theActive Sensing Array.

FIG. 57 shows an exploded view of embodiment where protrusions areaffixed to the Active Sensing Array.

FIG. 58A shows top view of dimensions used in the prototype embodimentof the Distinct Plate Matrix and Protrusion Matrix Layers Technique.

FIG. 58B shows side view of dimensions used in the prototype embodimentof the Distinct Plate Matrix and Protrusion Matrix Layers Technique.

FIG. 59 shows Plate alignment over Active Sensing array.

FIG. 60 shows top view of Rigid Plate properly aligned and inside ofcorresponding sensing elements on the Active Sensing array.

FIG. 61A shows top view of Plate Matrix.

FIG. 61B shows side view of Plate Matrix.

FIG. 62A shows top view of Protrusion Matrix.

FIG. 62B shows side view of Protrusion Matrix.

FIG. 63 shows Plate Matrix aligned with an Active Sensing Array.

FIG. 64 shows the top view of a protrusion properly aligned upon thecorresponding sensing element on the Active Sensing array.

FIGS. 65A-65F shows various valid and invalid configurations ofprotrusions.

FIGS. 66A-66C shows A Bottom, B Side, and C Top Views of thesuperposition of a properly aligned Plate Matrix and Protrusion Matrix.

FIG. 67 shows a cut out view of the superposition of a properly alignedPlate Matrix and Protrusion Matrix.

FIG. 68A shows a horizontal sensor, as on a table.

FIG. 68B shows a vertical sensor, as on a wall.

FIG. 69 shows an embodiment of an Integrated Plate and Protrusion Layer.

FIG. 70 shows a side view of an Integrated Plate and Protrusion Layerwith slits and rectangular protrusions.

FIG. 71 shows a side view of an Integrated Plate and Protrusion Layerwith slits and rectangular protrusions such that the protrusionscontinue through the junction to be flush with the plate.

FIG. 72 shows a side view of an Integrated Plate and Protrusion Layerwith slits and trapezoidal protrusions.

FIG. 73 shows a side view of an Integrated Plate and Protrusion Layerwith wider slits and rectangular protrusions.

FIG. 74 shows a top view of an Integrated Plate and Protrusion Layerwith slits that, at the junctions, are not flush with the outer surfaceof the plates.

FIG. 75 shows a top view of an Integrated Plate and Protrusion Layerwith slits and rectangular protrusions such that the protrusionscontinue through the junction to be flush with the plate.

FIG. 76 shows a top view of an Integrated Plate and Protrusion Layerwith wider slits that, at the junctions, are not flush with the outersurface of the plates.

FIGS. 77A-77C shows examples of sets of corner protrusions constitutinga protrusion over a sensing element.

FIG. 78 shows a side view of Flat Top Integrated Plate and ProtrusionLayer embodiment.

FIG. 79 shows the outer face of a Flat-Top Integrated Plate andProtrusion Layer embodiment.

FIG. 80 shows the inner face of a Flat-Top Integrated Plate andProtrusion Layer embodiment.

FIG. 81 shows a Flat Top Plate Matrix Layer.

FIG. 82 shows an Integrated Protrusion and Base Support Layer.

FIG. 83 shows an acceptably rigid plate.

FIG. 84 shows an acceptably semi-rigid plate.

FIG. 85 shows an unacceptably non-rigid plate.

FIG. 86 shows a cross Section of Force Distribution at a plate.

FIG. 87 shows a schematic view of an isolated plate and its mechanicallyinterpolated force distribution exclusively to adjacent sensingelements.

FIG. 88 shows the plate and protrusion dimensions used in the prototypeembodiment of the Integrated Plate and Protrusion Layer.

FIG. 89A shows photo-resistive ink pattern for plates.

FIG. 89B shows photo-resistive ink pattern for protrusions.

FIG. 90A shows cross section view the compression plates manufacturingembodiment.

FIG. 90B shows top view the compression plates manufacturing embodiment.

FIG. 91A shows an embodiment of a plate and protrusion layer with plateshaving discontinuous corner protrusions and abutting corners.

FIG. 91B shows an embodiment of a single part flat top plate andprotrusion layer with plates having discontinuous corner protrusions andabutting corners.

FIG. 92 shows an embodiment with the circuit board coplanar with theActive Sensing Array.

FIG. 93 shows an exploded view of an interior grid tile with bridgingplates.

FIG. 94 shows a top view of an interior grid tile with bridging plates.

FIG. 95 shows a side view of an interior grid tile with bridging plates.

FIG. 96A shows the alignment of the bridging plates of adjacent tiles.

FIG. 96B shows the correct positioning of the bridging plates ofadjacent tiles.

FIG. 97A shows side view of circuit board embedded in the base layer ofa tile with Bridging plates.

FIG. 97B shows bottom perspective view of circuit board embedded in thebase layer of a tile with Bridging plates.

FIG. 98A shows the schematic of adjacent tile alignment of tiles withbridging plates and assembly of circuitry under the support layer inposition.

FIG. 98B shows the alignment of adjacent tiles with bridging plates andassembly of circuitry under the support layer.

FIG. 99 shows schematic of a grid of tiles with bridging plates beingproperly aligned.

FIG. 100 shows of a grid of tiles with bridging plates in position.

FIG. 101 shows of a grid of tiles with bridging plates in position withbridging tiles transparent exposing bridge plate alignment onprotrusions.

FIG. 102 shows a grid of interior, north, east and northeast tilesembodiment.

FIG. 103 shows a schematic alignment of a 3×3 grid of interior, north,east and northeast tiles embodiment.

FIG. 104 shows a 3×3 grid of interior, north, east and northeast tilesembodiment in their proper positions.

FIG. 105 shows a deformable patch on a cylindrical surface.

FIG. 106 shows a deformable patch on a conic surface.

FIG. 107 shows the inside view of an assembly of a cylindrical sectioncurved sensor.

FIG. 108 shows the outside view of an assembly of a cylindrical sectioncurved sensor.

FIG. 109 shows a height edge view of a cylindrical section IntegratedPlate and Protrusion Layer.

FIG. 110 shows an outside view of a cylindrical section Integrated Plateand Protrusion Layer.

FIG. 111 shows an inside view of a cylindrical section Integrated Plateand Protrusion Layer.

FIG. 112 shows a sensor mounted on a cylindrical surface.

FIG. 113 shows a plate matrix of hexagonal plates.

FIG. 114 shows a protrusion matrix corresponding to a hexagonal platematrix.

FIG. 115 shows an Integrated Plate and Protrusion Layer with hexagonalplates.

FIG. 116 shows an Active Sensing Array with corresponding spacing to ahexagonal plate matrix.

FIG. 117 shows a Hexagonal Integrated Plate and Protrusion Layerpositioned above the Active Sensing Array.

FIG. 118 shows a hexagonal plate with corners labeled.

FIG. 119 shows an embodiment with the protrusions affixed to the activesensing array, which is wrapped around the support layer to circuitry onthe bottom of the tile.

FIG. 120 showing Connector Tails separated into banks of 16 trace lines.

FIG. 121 showing layers and applied force on the integrated protrusionand base layer embodiment.

FIG. 122 shows an embodiment with a touch device and two range imagingcameras.

FIG. 123 shows the left hand and right hand of one individual user.Beyond the individual user maximum reach, another individual user isidentified.

FIG. 124 shows a range imaging camera.

FIG. 125 shows a touch imaging device.

FIG. 126 shows a pressure imaging apparatus.

FIG. 127 shows a table top embodiment.

FIG. 128 shows a floor embodiment.

FIG. 129 shows an embodiment of the Touch-Range Fusion Apparatus with acomputer.

FIG. 130 shows the outline of a hand using edge detection, a skeletonmatched to edge hand, and finger touches identified.

FIG. 131 shows that cubes can be placed at the four corners.

FIG. 132 shows an embodiment of the Touch-Range Fusion Apparatus.

FIG. 133 shows an embodiment with a touch device, range imaging camera,and supporting stand for the range imaging camera.

FIG. 134 shows a Touch Device 101 with a set of Contact Points Pk.

FIG. 135 is a block diagram of Data from Range Imaging Camera and TouchDevice being processed by the computer and stored in computer memory.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings wherein like reference numerals refer tosimilar or identical parts throughout the several views, and morespecifically to FIGS. 35 and 36 thereof, there is shown an apparatus 1for sensing. The apparatus 1 comprises a computer 3. The apparatuscomprises two or more individual sensing tiles 2 in communication withthe computer 3 that form a sensor surface that detects force applied tothe surface and provides a signal corresponding to the force to thecomputer 3 which produces from the signal a time varying continuousimage of force applied to the surface, where the surface is contiguous,detected force can be sensed in a manner that is geometricallycontinuous and seamless on a surface.

The present invention pertains to a sensor 200, as shown in FIGS. 50-52.The sensor 200 comprises a grid 126 of wires 23 that defineintersections and areas of space between the wires 23. The sensorcomprises a set of protrusions 30 that are in contact with a pluralityof intersections of the grid 126 of wires 23, and a mechanical layerthat is disposed atop the set of protrusions 30, so that force impartedto the top of that mechanical layer is transmitted through theprotrusions 30, and thence to the. The sensor comprises a computer 3 incommunication with the grid 126 which causes prompting signals to besent to the grid 126 and reconstructs a continuous position of force onthe surface from interpolation based on data signals received from thegrid 126.

The sensor 200 may include a force resistive material in proximity to aplurality of the intersections of the grid 126 of wires 23. The forceresistive material may be disposed only in proximity to a plurality ofthe intersections of the grid 126 of wires 23 and in spacedrelationship.

The present invention pertains to a sensor. The sensor comprises acomputer 3 having N dual analog/digital I/O pins and M digital I/O pinsfor data, where M is less than N and M and N are positive integersgreater than three. The sensor comprises a pressure sensing array havingN rows and M columns, with the N I/O pins in communication with the Nrows and up to M columns in communication with the M I/O pins withoutusing any transistors or other switchable electronic components outsideof the computer 3.

The present invention pertains to a method for determining locations oftiles 2 of a sensor. The method comprises the steps of sending a querysignal from a computer 3 to at least a plurality of the tiles 2 incommunication with the computer 3 asking each of the plurality of tiles2 to identify at least one adjacent tile 2 that the tile 2 is inelectrical communication. There is the step of receiving by the computer3 responses to the query from the plurality of tiles 2. There is thestep of forming with the computer 3 from the responses a geometric mapof the tiles' locations relative to each other.

The present invention pertains to a method for sensing. The methodcomprises the steps of detecting a force applied to a sensor surfaceformed of two or more individual sensing tiles 2 from an object movingacross the surface where the surface is contiguous, detected force canbe sensed in a manner that is geometrically continuous and seamless on asurface. There is the step of providing a signal corresponding to theforce to a computer 3 from the tiles 2 in communication with thecomputer 3. There is the step of producing with the computer 3 from thesignal a time varying continuous image of force applied to the surface.There may be the step of connecting an additional tile 2 to at least oneof the two tiles 2 to expand the size of the sensor surface, where thesurface includes the additional tile 2 and is contiguous, detected forcecan be sensed in a manner that is geometrically continuous and seamlesson a surface.

The present invention pertains to a method for sensing. The methodcomprises the steps of imparting a force to a top of a mechanical layerthat is transmitted through to intersections defined by a grid 126 ofwires 23 having areas of space between the wires 23. There is the stepof causing prompting signals with a computer 3 in communication with thegrid 126 to be sent to the grid 126. There is the step of reconstructingwith the computer 3 a continuous position of the force on the surfacefrom interpolation based on data signals received from the grid 126.

The present invention pertains to a sensor 200. The sensor comprises agrid 126 of wires 23 that define intersections and areas of spacebetween the wires 23. The sensor comprises a set of protrusions 30 thatengage with a plurality of intersections of the grid 126 of wires 23,and an outer surface layer having an inner face that is in juxtapositionwith the set of protrusions 30 and an outer face, so that force impartedto the outer face of the outer surface layer is transmitted through theinner face of the outer surface layer to the protrusions 30 and theplurality of intersections. The sensor comprises a computer 3 incommunication with the grid 126 which causes prompting signals to besent to the grid 126 and reconstructs an antialiased image of force uponthe outer face of the outer surface layer from interpolation based ondata signals received from the grid 126.

The outer surface layer may be a mechanical layer, and the set ofprotrusions 30 are disposed between the grid 126 of wires 23 and themechanical layer. The grid 126 of wires 23 may be disposed between theset of protrusions 30 and the outer surface layer.

The present invention pertains to a method for sensing. The methodcomprises the steps of imparting a force to an outer face of an outersurface layer that is transmitted through an inner face of the outersurface layer to a set of protrusions 30 and a plurality ofintersections defined by a grid 126 of wires 23 having areas of spacebetween the wires 23. There is the step of causing prompting signalswith a computer 3 in communication with the grid 126 to be sent to thegrid 126. There is the step of reconstructing with the computer 3 anantialiased image of the force on the outer face of the outer surfacefrom interpolation based on data signals received from the grid 126.

The present invention pertains to a sensor 200. The sensor comprises agrid 126 of wires 23 that define intersections and areas of spacebetween the wires 23. The sensor comprises a set of protrusions 30 thatare in contact with a plurality of intersections of the grid 126 ofwires 23, and an outer surface layer having an inner face that isdisposed in contact with the grid 126 of wires 23 and an outer face, sothat force imparted onto the outer face of the outer surface layer istransmitted through the inner face of the outer surface layer to theprotrusions 30, and thence to the intersections of the grid 126 wires 23which are thereby compressed between the outer surface layer andprotrusions 30; and that the protrusions 30 thereby focus the impartedforce directly onto the intersections. The sensor comprises a computer 3in communication with the grid 126 which causes prompting signals to besent to the grid 126 and reconstructs an antialiased image of force uponthe outer face of outer surface layer from interpolation based on datasignals received from the grid 126.

The present invention pertains to a sensor 200. The sensor comprises agrid 126 of wires 23 that define intersections and areas of spacebetween the wires 23. The sensor comprises a set of protrusions 30 thatare in contact with a plurality of intersections of the grid 126 ofwires 23, and a mechanical layer having a plurality of plates 35 that isdisposed atop the grid 126 of wires 23, so that force imparted to thetop of the mechanical layer is transmitted through the intersections,and thence to the protrusions. The sensor comprises a computer 3 incommunication with the grid 126 which causes prompting signals to besent to the grid 126 and reconstructs a continuous position of force onthe surface from interpolation based on data signals received from thegrid 126.

The mechanical layer may include a flexible touch layer disposed on theplurality of plates 35. Each plate 35 may have corners 125 that arealigned over a corresponding protrusions 30 outer face.

The present invention pertains to a sensor 200. The sensor comprises agrid 126 of wires 23 that define intersections and areas of spacebetween the wires 23. The sensor comprises a set of protrusions 30 thatare in contact with a plurality of intersections of the grid 126 ofwires 23. The sensor comprises a plate layer having a plurality ofplates 35 that is disposed atop the grid 126 of wires 23. The sensorcomprises a flexible touch layer disposed on the plate layer, whereinforce imparted to the touch layer is transmitted through the plate layerand at least one protrusion to the intersections. The sensor comprises acomputer 3 in communication with the grid 126 which causes promptingsignals to be sent to the grid 126 and reconstructs a continuousposition of force on the surface from interpolation based on datasignals received from the grid 126.

The present invention pertains to a sensor 200. The sensor comprises agrid 126 of wires 23 that define intersections and areas of spacebetween the wires 23. The sensor comprises a set of protrusions 30 thatare in contact with a plurality of intersections of the grid 126 ofwires 23. The sensor comprises a plate layer having a plurality ofplates 35 that is disposed atop the grid 126 of wires 23. The sensorcomprises a flexible touch layer disposed on the plate layer, whereinforce imparted to the touch layer is transmitted through the plate layerto the intersections layer, and thence to the protrusions 30. The sensorcomprises a computer 3 in communication with the grid 126 which causesprompting signals to be sent to the grid 126 and reconstructs acontinuous position of force on the surface from interpolation based ondata signals received from the grid 126.

The present invention pertains to a sensor 200. The sensor comprises aset of plates 35 that are in contact from the bottom at their corners125 with a set of protrusions 30 that are in contact from above with aplurality of intersections, each having a sensing element, of the grid126 of wires 23, and a thin top surface layer 127 that is disposed atopthe grid 126 of plates 35, so that force imparted from above onto thetop surface layer 127 is transmitted to the plates 35 and thence to theprotrusions 30, and thence to the intersections of the grid 126 wires 23which are thereby compressed between the base 47 and protrusions 30; andthat the protrusions 30 above thereby focus the imparted force directlyonto the sensor intersections, as shown in FIG. 52. The sensor comprisesa computer 3 in communication with the sensor grid 126 which causesprompting signals to be sent to the grid 126 and reconstructs acontinuous position of force on the surface from interpolation based ondata signals received from the grid 126.

Each sensing element may include FSR 24. When force is imparted to thesurface layer, each protrusion may be aligned to be in contact with acorresponding sensing element 26. The sensor may include adhesive 40disposed between the surface layer and the set of plates 35, and betweenthe protrusions 30 and the grid 126, and between the grid 126 and thebase 47.

Each plate 35 may be positioned such that its corners 125 are alignedinside of the adjacent sensing elements 26. The plates 35 may bespecially aligned such that there is a gap between the plates 35, andthat a center of the gap between the corners 125 of the plates 35 isaligned to correspond with a sensing element 26. Each protrusion may bea rigid bump of plastic, metal, wood or glass and focuses force onto thecorresponding sensing element 26, each protrusion having a shape whosecontact with the corresponding sensing element 26 lies exactly upon orinside of the corresponding sensing element 26. The protrusions 30 maycontinue through the gap between the plates 35 to be flush with theplates 35. The protrusions 30 may emanate from vertices of the plates 35with the plates 35.

In regard to the surface layer in contact with the set of plates 35, andthe protrusions 30 in contact with the grid 126, and the grid 126 incontact with the base 47, it is understood that in contact also includesthe situation when adhesive 40 is between the surface layer and the setof plates 35, and adhesive 40 is between the protrusions 30 and the grid126, and adhesive 40 is between the grid 126 and the base 47.

The present invention pertains to a method for sensing. The methodcomprises the steps of imparting force from above onto a top surfacelayer 127 that is transmitted to a set of plates 35 and thence to a setof protrusions 30, and thence to a plurality intersections of a grid 126of wires 23 which are thereby compressed between the base 47 andprotrusions 30, where the set of plates 35 are in contact from theirbottom at their corners 125 with the set of protrusions 30 that are incontact from above with the plurality of intersections of the grid 126of wires 23 disposed on the base 47; and that the protrusions 30 abovethereby focus the imparted force directly onto the intersections. Thereis the step of causing prompting signals by a computer 3 incommunication with the grid 126 to be sent to the grid 126. There is thestep of reconstructing with the computer 3 a continuous position offorce on the surface from interpolation based on data signals receivedfrom the grid 126.

The present invention pertains to a sensor 200. The sensor comprises aset of protrusions 30 that are in contact from the bottom with aplurality of intersections of the grid 126 of wires 23, and a set ofplates 35 that are in contact from the top with a plurality ofintersections of the grid 126 of wires 23, and a thin top surface layer127 that is disposed atop the set of plates 35, so that force impartedfrom above onto the top surface layer 127 is transmitted to the plates35, and thence to the intersections of the grid 126 wires 23, and thencethe protrusions 30, which are thereby compressed between the plates 35and protrusions 30; and that the protrusions 30 underneath thereby focusthe imparted force directly onto the sensor intersections. The sensorcomprises a computer 3 in communication with the sensor grid 126 whichcauses prompting signals to be sent to the grid 126 and reconstructs acontinuous position of force on the surface from interpolation based ondata signals received from the grid 126.

There may be the step of imparting force to a top of a mechanical layerthat is transmitted through at least one intersection of a plurality ofintersections, and thence to at least one protrusion of a set ofprotrusions 30 in contact with at least one of the intersections, wherethe intersections are defined by a grid 126 of wires 23 and areas ofspace between the wires 23, and the mechanical layer has a plurality ofplates 35 that are disposed atop the grid 126 of wires 23.

There may be the step of imparting force to a top of a mechanical layerthat is transmitted through at least one protrusion of a set ofprotrusions 30 to at least one intersection of a plurality ofintersections, where the intersections are defined by a grid 126 ofwires 23 and areas of space between the wires 23, and the mechanicallayer has a plurality of plates 35 that are disposed atop the grid 126of wires 23.

The present invention pertains to an apparatus 104 for inputtinginformation into a computer 3, as shown in FIGS. 122-129. The apparatuscomprises a 3d sensor that senses 3d information and produces a 3doutput. The apparatus comprises a 2d sensor that senses 2d informationand produces a 2d output. The apparatus comprises a processing unitwhich receives the 2d and 3d output and produces a combined output thatis a function of the 2d and 3d output.

Objects may be identified and tracked in 3D and 2D by the 3D and 2Dsensors. Fingers, hands, feet, people, pens or other objects may beidentified and tracked in 3D and 2D. The apparatus may include a memoryand wherein the identity of each object is maintained over time. Theidentity of objects from the 3D sensor may be paired with objects fromthe 2D sensor by the processing unit. The 2D sensor has a surface andthe 2D sensor may sense contact on the surface. The 2D sensor may senseimposed force on the surface. The 2D sensor may include a pressureimaging sensor. The 3D sensor may include a range imaging camera. The 3Dsensor may include an IR depth camera. The 3D sensor may include an RGBcamera. The apparatus may include a display upon which the combinedoutput is displayed.

The present invention pertains to a method for inputting informationinto a computer 3. The method comprises the steps of producing a 3doutput with a 3d sensor that senses 3d information. There is the step ofproducing a 2d output with a 2d sensor that senses 2d information. Thereis the step of receiving the 2d and 3d output at a processing unit.There is the step of producing a combined output with the processingunit that is a function of the 2d and 3d output.

There may be the step of identifying and tracking objects in 3D and 2Dby the 3D and 2D sensors. There may be the step of identifying andtracking fingers, hands, feet, people, pens or other objects in 3D and2D. There may be the step of maintaining in a memory the identity ofeach object over time. There may be the step of pairing with theprocessing unit the identity of objects from the 3D sensor with objectsfrom the 2D sensor. There may be the step of the 2D sensor sensescontact on its surface. There may be the step of the 2D sensor sensesimposed force on its surface. The 2D sensor may include a pressureimaging sensor. The 3D sensor includes a range imaging camera. There maybe the step of displaying on a display the combined output.

The grid of conductive wires 126 is comprised of the conductive tracelines 23 on the outer and inner surface sheets 21. An intersection ofthe grid of wires 128 is the location where two conductive trace lines23 meet. The intersection is also where the FSR material 24 is located.The flexible touch layer 38 constitutes a top surface layer 127 for thepressure imaging apparatus 1 in the embodiments utilizing plates 35 andprotrusions 30.

The following is a description in regard to the operation of theinvention.

A List of Hardware Components:

Active Sensing Array: The Active Sensing Array 20 as seen in FIG. 1consists of two Sensor Surface Sheets 21 facing each other, with onerotated 90° with respect to the other, as seen in FIG. 2. Each of thetwo Sensor Surface Sheets 21 consists of the Non-Conductive SurfaceSubstrate 22 with printed Conductive Trace Lines 23 with small amountsof Force Sensitive Resistive (FSR) material 24 printed over them, asseen in FIG. 3 and in an exploded view in FIG. 4, at intervals such thatwhen the two Surface Sheets 21 are placed into mutual contact, with theinked sides facing each other, the FSR 24 material is place in thevicinity of the intersections of the grid of Conductive Trace Line 23 asseen in FIG. 1, but is not required at other locations of the sensorsurface.

A Description Explaining how the Tiles 2 are Connected Together:

The sensor tiles 2 are connected together by wiring and a physicallinking device in an apparatus 1 containing a plurality of adjacenttiles as shown in schematic in FIG. 38.

Wiring between tiles is used for the system protocol communication andto identify local tile neighbors. The protocol wiring depends on thetopology of the protocol used in the system. In one implementation, thetiles are connected together by an I²C hub. In this system, the wiringstarts at the master and reaches each sensor in the grid. To detect thelocal neighbors of each sensor, wires 23 are passed from one sensor tileto its neighbors.

In addition to wiring, a physical connector is used to link adjacenttiles. The appearance of this connector depends on the desired use ofthe system. In one implementation, as seen in FIG. 41, FIG. 42A and FIG.42B, a plastic connector 71, which has holes located at key positions,is placed between adjacent tiles 2. The holes on the connector 71 lineup with tabs 72 on the base support layer 32 of each tile 2. Theconnector can then slide onto the two adjacent devices and providesadditional support to the grid.

FIG. 41 shows an exploded view of the base layer 32 with tabs 72 and theconnector 71; FIG. 42A shows proper alignment of tabs 72 into connector71; FIG. 42B shows proper position of tabs 72 and connector 71 for twoadjacent tiles.

How Each Layer in the Profile View is Made, how the Overall Profile isMade, and the Purpose of Each Layer:

How Each Layer is Made:

The semi-rigid touch Layer 31 and the protrusions 30 as seen in FIG. 15,can be a single mechanical component, which can be made of plastic,glass, wood, metal, or any other semi-rigid material. This component canbe manufactured by a variety of standard methods, including injectionmolding, stamping, and cold casting.

In an alternate embodiment, as seen in FIG. 12, the protrusions 30 canbe rigidly affixed to surface substrate 22 of the outer sensor surfacesheet 21 at the corresponding sensing element locations. One method fordoing this is by cold casting: In one method of manufacture, a mold,which can consist of silicone rubber, that contains regularly spacedholes, is placed atop the outer side of surface substrate 22, and aresin is poured into these holes. When the resin hardens, the mold isremoved, and the resin forms regularly spaced bumps upon the top surfaceof the surface substrate 22. In this embodiment, touch layer 31 issimply a semi-rigid sheet, which can be made of plastic, glass, wood ormetal, or any other semi-rigid material. One advantage of this alternateembodiment is that it ensures that the protrusions 30 remain correctlyaligned with the FSR material 24 corresponding to the active area ofeach sensing element 27 during operation of the sensor. Such aconstruction constitutes an active sensing array with attachedprotrusions 55.

How the Overall Profile is Made:

The overall profile is made by assembling the component layers duringthe manufacturing process.

For clarity, ‘Outer’ or ‘Outer Surface’ of a component, is designated tosignify the side/direction of the device from which the external forceis being applied, such as a user touching the surface. ‘Inner’ or ‘InnerSurface’ is designated to be the opposite direction of Outer.

The Purpose of Each Layer from Outer to Inner, as Seen as a Sensor CrossSection in FIG. 12, Where Outer to Inner in this Case is from the Top ofthe Page Downward:

The purpose of the semi-rigid touch layer 31 and the protrusions 30, asseen in FIGS. 12 and 13, is to redistribute continuous force 34 which isapplied to the outer surface of the semi-rigid touch layer 31 so thatall applied force is distributed only to the active sensing elementareas 27, namely at the outer or inner surface at the junctions ofconductor traces 23 in the active sensing array 20, as seen in FIG. 11.

The next inner layer is the non-conductive sensor substrate 22 of theouter sensor surface sheet 21 of the active sensing array 20, which canbe made of thin acetate which can, in one implementation be 5 mils inthickness, followed by the next inner layer of a the pattern ofmetal-infused ink conducting trace lines 23 which is printed on theinner side of the substrate 22.

The next inner layer shows FSR material 24 against FSR material 24: Theouter FSR 24 pattern that is overprinted over the conducting lines 23 ofthe outer sensor surface sheet 21 of the active sensing array 20, asshown in FIGS. 3 and 4. The inner FSR 24 is overprinted over theconducting lines 23, the next inner layer, of the inner sensor surfacesheet 21 of the active sensing array 20. In operation, these two FSR 24components are in contact with each other, but are not mechanicallyaffixed to each other.

The next inner layer is the non-conductive sensor substrate 22 of theinner sensor surface sheet 21 of the active sensing array 20, which canbe made of thin acetate which can, in one implementation be 5 mils inthickness, together with the pattern of metal-infused ink conductingtrace lines 23 of the previous layer, which is printed on the outer sideof this substrate 22.

The next inner layer is the support layer 32 which can be made of anysolid material, such as glass, acrylic, wood or metal. In oneimplementation, it was made of ¼″ thick acrylic.

For clarity, the sensing element 26 comprises all the material on all ofthe Active Sensing Array 20 at the junction of conductor traces 23enabling the electronically measuring force in that region, as seen inFIG. 10. The active area of a sensing element 27 corresponds to theinner or outer area on the surface of the active sensing array 20corresponding to that location of that sensing element, specificallywhere force is focused upon, as seen in FIG. 11. As such, ‘in contactwith the sensing element’ implies contact with the active areacorresponding to that sensing element.

A detailed description of following a signal through each feature of theinvention from start to finish: Specifically, how the signal isgenerated from an object contacting the outer surface of the touch layerand what happens to it from that point on through the conducting lines,along the network, and ultimately to the computer 3 where it is imaged,covering every specific step along the way, including how interpolationis applied to the signal as part of this detailed description followingthe signal.

FIG. 13 shows the imposition of force or pressure 34 applied to thesemi-rigid upper plate being mechanically transmitted to nearbysupporting protrusions 30, and thence to the pressure sensing activearea of the sensing elements 27 where conducting lines 23 intersect onactive sensing array 20 of the tile. In this embodiment the protrusionsare attached to the outer surface of the active sensing array 20, ratherthan to the semi-rigid touch layer 31.

The nearby protrusions 30 and corresponding sensing elements 26 do notneed to be on the same tile, but rather can be on adjacent, mechanicallyseparate tiles, as in FIG. 14.

FIG. 14 shows the imposition of force or pressure 34 applied to thesemi-rigid upper plate being mechanically transmitted to nearbysupporting protrusions 30 on two adjacent but mechanically distincttiles, and thence to the pressure sensing active area of the sensingelements 27 where conducting lines 23 intersect on respective activesensing arrays 20 of distinct sensor tiles. In this embodiment theprotrusions 30 are attached to the outer side of the active sensingarray 20, rather than to the semi-rigid touch layer 31.

Interpolation

For each sensor apparatus, force imparted on a surface is mechanicallyredistributed such that all the force is focused onto a grid of pressuremeasuring sensing elements 26 under that surface on one or a pluralityof tiles 2 containing active sensing arrays 20 containing sensingelements 26, as demonstrated the various embodiments described herewithin. Interpolation is achieved by this mechanical redistribution.When contact is made upon the outer surface of the apparatus and above asensing element 26, the force applied to that location is registered atsensing element 26. When the contact is moved between locations abovesensing elements 26, the force is applied to multiple sensing elements26. The distribution of the force of the contact at each of the sensingelements 26 is used to calculate the centroid of that contact.

In particular, consider the 2×2 array of adjoining sensing elements 26at respective locations (i,j), (i+1,j), (i,j+1), (i+1,j+1). Theseintersections may be labeled, A, B, C, and D as seen in FIG. 48, wherethe intersections represent the locations of sensing elements 26 on anactive sensing array 20. The forces sensed at each of these sensingelements 26 may be described by fA, fB, fC, and fD, respectively.

Because the mechanical redistribution of force described here within isapproximately linearly as a function of position, the centroid position[x, y] of the touch can be well approximated by the following linearinterpolation of position as a function of force at the four locations.One may first approximate the fractional east/west position of thecentroid between two adjoining columns by linear interpolation followedby a compensation for any nonlinearity:

u′=(fB+fD)/(fA+fB+fC+fD)

u=COMP(u′)

and the fractional north/south position between two adjoining rows bylinear interpolation followed by a compensation for any nonlinearity:

v′=(fC+fD)/(fA+fB+fC+fD)

v=COMP(v′)

Interpolation of touch position between rows and columns is based onrelative force at the nearest row/column intersections A, B, C and D asseen in FIG. 48 and described above. From this information, the centroidposition of any single touch within the sensor array can be computed.

One can make use of a compensation function, represented in the aboveequations by the function COMP( ). This function is a monotonic mappingfrom the domain 0 . . . 1 to the range 0 . . . 1. This functioncompensates for non-linearity in the mechanical interpolation of thesensor between successive sensor elements. For example, a pressureapplied to a location that is 0.25 of the way from the left neighboringconductor line 23 for a sensing element 26 to the right neighboringconductor line 23 of a neighboring sensing element 26 will result in aproportional value of pressure, with respect to total pressure, downonto the protrusion 30 on the right which is greater than 0.0 and lessthan 0.5, but which is not necessarily 0.25. The use of a compensationfunction corrects for any disparity.

FIG. 49 shows a typical set of values for the compensation function. 91is the fractional proportion u′ from left to right of the sensedpressure, in the range from 0 to 1. 92 is the desired proportionalgeometric position. 93 is the function that maps 91 to 92.

In another embodiment, even more precise compensation can be attained bydefining two compensation functions: COMP_u(u′, v′) and COMP_v(u′, v′).In all implementations, the compensation values can be constructed by astandard calibration procedure in which pressure is applied at knownpositions on the sensor, and the results stored in a table. A continuouscurve, such as a piecewise linear or piecewise cubic function, is thenfit between measured values from this table, to create a continuousfunction. In the case of COMP_u and COMP_v, the table is twodimensional, and the interpolation between table values is effected by acontinuous two dimensional function, such as piecewise bilinear orpiecewise bicubic.

From the values of u and v, the coordinates of the centroid may beobtained:

[x,y]=[S*(i+u),S*(j+v)]

where S is the spacing between successive rows and columns in the sensorarray. In one embodiment, S=⅜″.

Scanning

One microcontroller is associated with each sensor tile. For each sensortile, that tile's microcontroller scans successive row/column pairswithin a sub-region. The microcontroller uses digital and analog I/Opins on the micro-controller to scan the sensor for pressureinformation. When connected, the sets of row and column wires 23 areeither assigned to be output or input wires 23. Output wires 23 canprovide a positive voltage or be set to ground. Input wires 23 caneither be set to ground or read a voltage from a wire. At the start ofeach frame, one output wire is set to a positive voltage, while the restof the output wires 23 are set to ground. The input wires 23 are alsoset to ground, except for one wire which scans the voltage coming fromthe intersection of the output and input wires 23. The firmware thenscans the next input wire, while setting the others to ground. After allinput wires 23 have been scanned, the next output wire is set to apositive voltage, while the first is set to ground, and the input wires23 are scanned again. This is repeated for all the voltage wires 23,until every intersection has been scanned.

Scanning the device gives a frame of pressure information whichregisters the fingers or other objects that imposed force upon the MFRL.On each sensor tile, the tile's microcontroller optionally compressesthe gathered sensor image data by ignoring all data values below achosen cut-off threshold (i.e.: this data is redefined to be identicallyzero). Non-zero data is formed in packets, using a compression techniquesuch as run-length encoding.

Communication from Tiles to the Computer 3

Data packets, each tagged with the identity of the tile from which itoriginated, are sent over a common data communication protocol that isshared by the microcontrollers of all of the tiles in the sensor array.One sensor tile is designated as the master tile 7. This master tile 7possesses a USB or similar communication connection 9 to the hostcomputer 3, as seen in FIG. 38. The master tile 7 sends all of thecompressed packets to the host computer 3.

On the host computer, the packets are assembled into a single seamlessimage of pressure.

Possible Applications for the Invention:

Electronic white boards.

Pressure sensitive floors. One use in this area is security, such as atan airport. In this application, the sensor array would be used inconjunction with image recognition software that can identify differentindividuals by the differing pressure patterns of their footsteps.

Pressure sensitive touch walls.

Pressure sensitive tables or desks.

Pressure sensitive surfaces for factories.

Pressure sensitive roadways, such as highways or bridges. Uses for thisinclude traffic monitoring, including both speed and weight of vehicles,as well as an accurate image of number of wheels and wheel distribution,which can be used for accurate assessment and categorization of vehicletype.

Pressure sensitive seats. Uses for this include train seats, automobileseats, airplane seats and seats for assistive devices such aswheelchairs.

Pressure sensitive displays. OLED displays as part of the touch layer.

Enabling Information about the Third Invention that has to do withMatching the Number of Lines to the Computer:

A given microcontroller chip has a particular number N of dualanalog/digital JO pins, while the number of purely digital IO pins 82 onthe microcontroller chip M. By connecting the N dual analog/digital IOpins 81 to N rows of an active sensing array 20, and up to M of thepurely digital IO pins 82 to the N columns of the active sensing array20, an active sensing array 20 driven from a single microcontroller canachieve up to N×M pressure sensing elements 26 without the requirementof supplementary electronic components. This architecture results in asimple configuration of electronic components.

One embodiment uses the PIC24HJ256GP610 microcontroller from MicroChip,which contains 84 data pins altogether, of which 32 are dualanalog/digital JO pins 81, and these can be used as analog voltage inputpins, one for each row of the sensor array. Setting aside the pins thatare used for external communication with other microcontrollers in thegrid of tiles, at least 32 digital JO pins 82 are available aspower/ground switchable pins to drive 32 columns of the sensing array.Thus, this particular microcontroller is able to drive a 32×32 arraypressure sensing tile 2, with no other electronics required aboard thetile other than a small number of resistors and capacitors to smooth outcurrent and avoid spikes in current.

The master tile 7 in this embodiment requires a single 3.3 voltregulator, such as the Fairchild REG1117A, to drive the 5V from the hostcomputer's USB port to the 3.3 volts required by the microcontroller. Noother electronics are required.

Utility of the Invention

There is currently no solution for low cost pressure sensing that can beeasily mass-produced and that is economically scalable to form aseamless surface of arbitrarily large surface area. There are indeedspecialized technologies, such as the UnMousePad by [Rosenberg] andTekScan, Inc. devices based on sensing grids that make use of forcesensitive resistance (FSR) materials 24 [Eventoff], but none of theseare designed or engineered to scale up reliably to large surface area atlow cost per unit sensing area.

The current invention is an inexpensive and flexible way to convert anyarea of floor, wall, table or any other surface into a “video camera forpressure” or pressure imaging apparatus. Once the apparatus 1 isconnected via a standard method for transferring digital signals, suchas a serial protocol over a USB cable, to a host computer 3, then thetime-varying pressure image of any and all touches upon the surface canbe read and processed to support many different applications.

The system consists of a set of one or more modular pressure tiles 2.These tiles 2 can be of arbitrary shape, including triangular, hexagonalor other irregular tile shape. In one embodiment, each tile 2 is asquare containing 32×32 sensing elements, so the total resolution of thesensing array will be the number of tiles times 32×32.

A networked tile assembly 18 is composed of a collection of tiles whichcommunicate with each other such that the physical arrangement of tilescan be reconstructed virtually. In one embodiment the size of each tileis 12 inches×12 inches square pressure tile 2 (though the sizes of tilesin an assembly need not necessarily be equivalent). In this embodiment,if every tile has 32×32 sensing elements 26, then the spacing betweensuccessive sensing elements is ⅜″.

Tiles can be assembled together to create an arbitrarily large seamlesspressure imaging apparatus 1. The apparatus 1 sends to a host computer 3a single time-varying composite image of pressure variation across theentire surface.

Power can optionally be augmented or supplied by one or moresupplementary power modules as needed.

The sensor can incorporate, on its upper surface, a mechanical forceredistribution layer that distributes pressure predictably so that thesensed pressure is well distributed to the sensing elements in the tile.

Step by Step Description of User Experience:

From the user's perspective, operation is as follows and as seen in FIG.35.

In one time step, the user imposes a finger or other physical object 34onto the top of the pressure sensing apparatus 1. A continuous image ofthis imposed pressure is transmitted by the pressure sensing apparatus 1to a host computer 3.

On the host computer 3 this image of spatially varying pressure isstored in a region of computer memory. From there, computer software onthe host computer 1 can be used to store the image to secondary storagesuch as a disk file, to display the image as a visual image on acomputer display 6, to perform analysis such as finger tracking, regionfinding, shape analysis or any other image analysis process which isstandard in the art, or for any other purpose for which an image can beused.

On the next time step, the above process is repeated, and so on for eachsuccessive time step.

Step by Step Description of Internal Working:

Internal operation begins when fingers or other objects 34 imposedownward force upon the outer surface of the semi-rigid touch layer 31,as seen in FIG. 13.

This force is then transmitted, and properly redistributed, from thesemi-rigid touch layer 31 to the sensing elements 26 on the activesensing array 30 of each sensor tile 2, as seen in FIG. 22. Onemicrocontroller 5 is associated with the tile circuit board 4 for eachsensor tile 2, as seen in FIG. 32. Grids of tiles 2 are physically, aswell as with electronic cabling 10, connected to form a coherent sensingapparatus 1, as seen in FIG. 36.

Then, for each sensor tile 2, that tile's microcontroller 5 scans thepressure values at the sensing elements at each successive row/columnpairs within a sub-region as described here within to form an image ofpressure.

On each sensor tile 2, the tile's microcontroller optionally compressesthe gathered sensor image data by ignoring all data values below achosen cut-off threshold (i.e.: this data is redefined to be identicallyzero). Non-zero data is formed in packets, using a compression techniquesuch as run-length encoding.

The packets, each tagged with the identity of the tile from which itoriginated, are sent over a common data bus that is shared by themicrocontrollers of all of the tiles 2 in the sensing apparatus 1 grid,as seen in FIG. 37. One sensor tile is designated as the hostcommunicator tile 7. This tile possesses a USB or similar communicationconnection 9 to the host computer 3. The host connection tile 7 sendsall of the compressed packets to the host computer 3, as seen in FIG.36.

On the host computer 3, the packets are assembled into a single image ofpressure. The identification of each tile, stored with each packet,together with pre-stored information about the relative position of eachtile, as seen in one organization of Tiles seen in FIG. 38 in thecorresponding Sample Tile Topology Table (below) is used by the hostcomputer 3 to place each sub-image in its proper location within thecomplete multi-tile image of pressure.

Sample Tile Topology Table, corresponding to FIG. 38

Tile ID Row Column T-0 0 0 T-1 0 1 T-2 0 2 T-3 0 3 T-4 1 0 T-5 1 1 T-6 12 T-7 1 3 T-8 2 0 T-9 2 1 T-10 2 2 T-11 2 3

Optionally, a protocol between the microcontrollers associated with eachtile can identify neighbor information within the tile grid itself. Inthis option, upon initialization of the connection between the tile gridand the host computer, each microcontroller is directed to send a datapacket through the shared bus which identifies all neighbors with whomit is connected, as well as the direction and Tile ID of that neighbor(north, east, west or south), as seen in FIG. 40. In FIG. 40 and theSample Tile Topology Table and Sample Tile Adjacency Table (below), theTile IDs are designated T-0, T-1, etc. The host computer stores thisinformation in a table, which is indexed by tile ID, seen in TileTopology Table (below). Each table entry contains a list of between oneand four neighbor ids for that tile in the respective North, South,East, and/or west column. As with the earlier described embodiment wherethe tile adjacency table is manually configured, the host computer 3uses this connectivity information to assemble all received data packetsinto the coherently reconstructed measured pressure data image withsensing element data from all sensing elements on all tiles in thefollowing manner: At each time step, starting with the location of theHost Tile, placing the pressure data for host tile in a particular blockof memory corresponding the data measured from that tile's sensingelements, then placing the data for the neighbors of the host tile intheir proper relative positions to the host tile, then placing data forthose neighbors in their respective relative positions, and so on, in abreadth first traversal of the entire connectivity graph, until data forall tiles has been properly placed in their respective positions on aTile Topology table. An advantage of this approach is that it allowsarbitrary arrangements of tiles to be accommodated.

The above method relies upon each processor knowing the identities ofits immediate neighbors. In one embodiment, processors determine theseidentities at initialization time as follows: (1) a neighbor-determiningsignal is sent from the host computer along the shared bus to eachtile's microcontroller in turn. A microcontroller only acts upon theneighbor-determining signal when that signal is addressed to its ownunique identity; (2) upon receiving this signal, the processor sends, inturn, an identity query to each of its immediate North, South, East andWest neighbors. (3) When a processor receives such an identity queryfrom a neighboring processor, it outputs its own identity through theshared bus to the host computer, which stores this neighbor informationinto a software table, such the Tile Adjacency Table below. In this way,the host computer is able to establish the identities of all immediateneighbors of all tiles.

Sample Tile Adjacency Table showing results of tile neighbor queries,corresponding to FIG. 38

Tile ID North South East West T-0 none T-4 T-1 none T-1 none T-5 T-2 T-0T-2 none T-6 T-3 T-1 T-3 none T-7 none T-2 T-4 T-0 T-8 T-5 none T-5 T-1T-9 T-6 T-4 T-6 T-2 T-10 T-7 T-5 T-7 T-3 T-11 none T-6 T-8 T-4 none T-9none T-9 T-5 none T-10 T-8 T-10 T-6 none T-11 T-9 T-11 T-7 none noneT-10

Tiles Seamlessly Abutting to Create a Seamless Pressure Sensing Device

The difficulty of seamlessly tiling sensor arrays can be described byanalogy with LCD arrays. When a collection of LCD monitors are arrayedto create larger image, there is generally a visible gap betweensuccessive monitors. This gap is caused by the fact that there are edgeconnections and electronics, outside of the image area of each monitor,which takes up non-zero area. Existing FSR based pressure sensor arrays,such as the TekScan sensor array, suffer from the same problem—thenon-zero area around the active sensing area which is taken up byconnectors and electronic components creates a geometric gap. Because ofthis gap, a plurality of TekScan sensors cannot be tiled to create aseamless larger sensing surface.

A plurality of TouchCo sensors cannot be seamlessly tiled for adifferent reason: Because the method of the TouchCo sensor requiresspatial interpolation upon a continuous area of FSR material betweensuccessive active conducting lines, the sensor cannot seamlesslyinterpolate in any area that is not between successive conducting lineson a single sensor array. Therefore, the sensor cannot seamlesslyinterpolate across different physical sensors.

Our method makes use of a mechanical interpolation layer that is able tospan physically different tiles. Therefore one of the novel features ofthe technique here is the ability to seamlessly interpolate detectedforce even between physically distinct sensing array tiles.

The Mechanism for Even Force Redistribution from the Continuous UpperTouch Layer to the Discrete Sensor Layer:

A mechanical layer is imposed on top of the active sensing array 20. Thepurpose of this layer is to redistribute the force imposed downward ontothe mechanical layer, so that all of this force is transmittedexclusively to the active areas of the surface of the active sensingarray 20, Where “active area” 27 is defined as any region in which theupper and lower conductive wires 23 cross, with FSR material 24sandwiched between them where they cross, as seen in FIGS. 10 and 11. Inparticular, every such intersection corresponds to a sensing element 26for measuring pressure data.

For clarity, the sensing element 26 comprises all the material on all ofthe Active Sensing Array 20 at the junction of conductor traces 23enabling the electronically measuring force in that region. The activearea of a sensing element 27 corresponds to the inner or outer area onthe surface of the active sensing array 20 corresponding to thatlocation of that sensing element, specifically where force is focusedupon. As such, ‘in contact with the sensing element’ implies contactwith the active area corresponding to that sensing element.

In one implementation, as seen in FIG. 16, the semi-rigid touch layer 31and protrusions 30 are constructed as a single part, implemented as athin semi-rigid plastic sheet with small raised bumps on its underside.The protrusions 30 are spaced so that when the this part is resting overthe active sensing array 20, each of these protrusions 30 sits over oneof the active areas of the corresponding sensing elements 27, namely thesmall regions of the tile where conductive trace lines 23 cross, withFSR layers 24 sandwiched between them, as seen in FIG. 17. FIG. 16 showsa semi-rigid touch surface with protrusions 33.

This structure forms a mechanism whereby a continuous change in positionof a touch on the outer side of the touch surface results in acorresponding continuous change in the relative force applied to thesensor junctions that are nearest to that touch. Those relative forces,when sent to the host computer as part of the data image, permit thehost computer to reconstruct the position of the touch through simplearithmetic interpolation.

FIG. 15 and FIG. 17 show a schematic profile view of the semi-rigidtouch surface with protrusions 33 sitting atop the active sensing array20. In this implementation, the bumps 30 are rigidly affixed to thesemi-rigid flat touch layer 31 as a coherent part 33. This part 33 sitsatop the non conduction substrate 21 of the active sensing array 20,which consists of an upper surface 21, a lower surface 21, each of whichincludes a respective FSR layer 24. In this figure, the conductive tracelines 23 of the active sensing array 20 are not shown. On the inner mostlayer is a solid support layer 32 providing a rigid base for theapparatus to counter the surface forces. In one embodiment, the supportlayer 32 can be a ½″ plate of acrylic.

In FIG. 17, it can be seen that the protrusions 30 contacts the uppersurface of the sensor tile only in the active regions 27 of the activesensing array 20.

This method of redistributing pressure also works when adjacent sensorelements are on physically disjoint adjacent tiles, as shown in FIG. 18.In FIG. 18, the constituent layers of the respective tiles are the sameas described above for FIGS. 15 and 17. FIG. 18 shows the semi-rigidtouch and protrusions layer 33 as a continuous sheet spanning theplurality of tiles, showing the physical redistributing pressure 34between sensor elements that belong to different physical sensor arraytiles.

FIG. 18 also shows an embodiment in which the active sensing array 20wraps around the edge of one of the tiles to connect the connector tails25 lines of that tile to the tile's printed circuit board 4, which arelocated on the underside of the support layer 32.

FIG. 18 illustrates seamless sensing across adjacent physical tiles, byusing mechanical force redistribution, as in the semi-rigid touch andprotrusion part 33 in this embodiment, distribute force between adjacentsensing elements on distinct tiles in a way that does not require amechanical connection between the underlying tiles themselves. When thetile array is in operation, there is no difference in signal responsebetween the following two cases: (a) adjacent sensing elements that areon the same physical tile, and (b) adjacent sensing elements that are ondifferent, but adjoining, physical tiles.

During any given time step, when a force is applied at the seam betweentwo adjoining tiles, some of the force is distributed to, as seen in thecross sectional view in FIG. 18 of one embodiment, the rightmost bump ofthe semi-rigid touch and protrusion layer 33 that touches the tile tothe left, and the remainder of the force is distributed to the leftmostbump of 33 that touches the tile to the right.

These two respective force signals will be detected by the respectivemicrocontrollers of the tile to the left and the tile to the right, andwill be sent by each of those tiles to the host computer as part of thattile's respective force image for this time step.

The host computer will then be able to reconstruct—from the respectivevalues along the rightmost edge of the force image from the tile on theleft and along the leftmost edge of the force image from the tile on theright—the position of the force applied in the region between the twoadjoining tiles, using exactly the same linear interpolation that isused to compute the position of a force applied between two conductinglines within a single tile.

The result is that from the perspective of the end user and softwareapplication developer, it makes no difference whether a touch upon thegrid of sensor array tiles falls within a single tile or between twoadjoining tiles of the grid.

Physical Implementation of the Active Sensor Array

In one embodiment, the Conductive Trace Lines 23 are printed with metalinfused ink over a non-conducting substrate 22, such as plastic, asshown in FIG. 3. All tracings 23 can be the same line width, the routingof traces 23 continue to form a Connector Tail 25 for connection to thetile's circuit board 4, with the tails possibly of a different/thinnerline width. In one embodiment of a tile, the Connector Tail 25 to thetile's printed circuit board 4 can be folded to the underside of thetile, around the protrusion 31 and Support Layer 32, with the circuitboard 4 placed beneath the Active Sensing Array 20, as seen in FIGS. 33and 34. This arrangement permits adjacent tiles to abut smoothly, withno gaps in sensing area between adjacent tiles, as seen in FIG. 18.

One embodiment of printed electrical conductor tracing lines 23 for thesurface sheet 21 of the Active Sensing Array 20 of the invention as inthe schematic on FIG. 5, all conducting lines 23 are 0.5 mm in width,and are spaced at intervals of ⅜″, and the line width of the connectortails 25, are 0.25 mm.

The FSR Ink 24 is printed as a grid of 1 mm squares over the Conductivelines 23 in an arrangement as shown in FIG. 6 resulting in a sensorsurface sheet 21, as seen in FIG. 3.

Note that FSR ink 24 need only be printed in the immediate neighborhoodof those parts of the sensor where conducting lines cross between topand bottom layer as seen in FIGS. 3, 10 and 11. This arrangement resultsin a very small usage of FSR per unit area.

FIG. 6 shows one embodiment the FSR layer 24 that is printed over theconducting lines 23 on the Sensor Surfaces 21 of the Active SensingArray 20 of the invention. In this embodiment, all conducting lines 23are 0.5 mm in width, and are spaced at intervals of ⅜″. Therefore, each1 mm square of printed FSR 24 is a patch that is slightly larger than0.5 mm×0.5 mm square of the intersections of the conducting lines 23 asseen in the exploded view in FIG. 10, so that the regions whereconducting lines cross are completely covered by FSR material, as seenin FIG. 11, with the active area of the sensing element 27 at that gridlocation shown as hatched.

FIG. 2 shows the exploded view of the superposition of conducting lines23 for top and bottom Sensor Surface Sheets 21 one Active Sensing Array20 of a tile, in their final operational positions. In one embodiment,all conducting lines are 0.5 mm in width, and are spaced at intervals of⅜″. FIG. 1 shows the Connector Tails 25 for connecting to the TileCircuit Board have not yet been folded under the tile. Therefore, theseConnector Tails 25 appear to stick out at a vertical and a horizontaledge.

In order to test the optimal conductor line 23 width, the technique hereincludes a testing procedure, a Test Active sensing array 20 ismanufactured where a Test Sensor Surface 21 is printed in which thethickness of the conducting lines 23 is varied between rows (andtherefore, on the obverse side of the sensor, of columns) as in FIG. 9A.This testing version of the active sensing array 20, as shown in FIG.9B, allows for selecting the optimal choice of line width for any givenapplication in final manufactured tiles. FIG. 9B shows the lineConductive Trace Lines 23 (with top and bottom Sensor Surfaces 21juxtaposed).

FIG. 8 shows the test pattern of the resistive ink 24 pattern printed onSensor Surface Sheet 21, for the testing embodiment of an active sensingarray 20 with graduated conducting trace line widths, used to test theoptimal conducting trace line 23 width, as seen in FIG. 9A. FIG. 9Bshows a superposition of the Sensor Sheets 21, in their finaloperational positions of the conducting lines 23 for the top surface 21of the active sensing array 20 and the conducting lines 23 for thebottom surface 21 of the test active sensing array 20, for a singletile.

How the top and the bottom ink pattern can be the same, merely rotated90 degrees and flipped over:

In one embodiment of the present invention pattern for the Trace Lines23, in which the Active Sensing Array 20 area is square, the top halfand bottom Sensor Sheet 21 of the Active Sensing Array 20 for the Tile 2are exactly the same. The bottom Sensor Sheet 21 is rotated 90° and thenflipped over, with respect to the Sensor Sheet 21. When this is done,the junctions and printed FSR 24 line up with each other exactly as seenin FIG. 2.

Electronic components printed and/or assembled directly onto the sensorarray:

Rather than requiring a separate Printed Circuit Board (PCB), allelectronics can, in one embodiment, be printed and/or assembled directlyonto the active sensing array 20, thereby greatly reducing the cost andcomplexity of manufacture.

Force Sensitive Resistors (FSR):

Force-sensing resistors consist of a semi-conductive material whichchanges resistance following the application of force to the surface.FSR generally consists of electrically conductive and non-conductiveparticles causing the material to be semi-conductive. FSR is normallysupplied as a sheet or as ink which can be applied using s screenprinting process. FSR is low cost and durable.

Firmware

For each group of tiles, there are three types of firmware: a slave anda master and host communication. The slave firmware is placed on themicro-controller for each sensor tile and is used to gather pressureinformation for that sensor. The master firmware is installed on atleast one micro-controller and manages the communication between thegroup of tiles and the host communication firmware transmits thepressure data to the computer.

Slave Firmware

The slave firmware uses digital and analog I/O pins on themicro-controller to scan the sensor for pressure information. Whenconnected, the sets of row and column wires are either assigned to beoutput or input wires. Output wires can provide a positive voltage or beset to ground. Input wires can either be set to ground or read a voltagefrom a wire. At the start of each frame, one output wire is set to apositive voltage, while the rest of the output wires are set to ground.The input wires are also set to ground, except for one wire which scansthe voltage coming from the intersection of the output and input wires.The firmware then scans the next input wire, while setting the others toground. After all input wires have been scanned, the next output wire isset to a positive voltage, while the first is set to ground, and theinput wires are scanned again. This is repeated for all the voltagewires, until every intersection has been scanned.

In one embodiment, 32 column wires are attached to digital I/O pins and32 row wires are attached to additional digital I/O pins that can readdifferent voltage levels. Using slave firmware algorithm gives a 32 by32 array of sensing element data with 4096 levels of pressure at eachintersection.

Master Firmware

The master firmware handles the flow of information from the individualtiles to other master tiles or to the computer. To get the pressureframe information from each tile, a communication protocol isestablished between the master and slave microchips. The protocoltopology varies depending on the size, shape and desired behavior of thetile grouping. In the communication protocol, data can either be polledby or streamed to the master micro-controller. In a polling system, themaster requests frames from individual tiles, managing the flow of datato the master tile. In a streaming system, the sensors attempt to streamits data to the master until the data has been received. The data passedto the master controller can represent the entire frame of data or canbe compressed. In one case, run-length encoding reduces the size of theframe by removing repeated zeros. Another form of compression involvessending only the difference between two frames. By sending only thedifference between frames, static objects on the sensor having no changein pressure signature do not require the sending of any continuous datato the master about those regions.

In one implementation, an I²C hub protocol is established betweenmultiple tiles. Information is sent from each of the slavemicro-controllers on a slave Tile 11 to a master micro-controller onMaster Tile 7. In FIG. 37, a schematic for an I²C hub is shown whichuses a Serial Data Line (SDA) 96, which transmits the data between theslaves and the master, and a Serial Clock (SCL) 97, which keeps time,and the power or Vdd 98.

In another implementation, the tiles can use an RS-485 communicationprotocol and be linked together in a daisy-chain multipoint setup. FIG.38 shows a rectangle grid of slave tiles 11 is connected in adaisy-chained S-pattern to a terminal Master Tile 7. The Master Tile 7,acting as the Host Communicator Tile 12, connects with an externalcomputer 3 over USB 9.

The accumulated pressure data is then passed through an additioncommunication protocol to the requesting device. In one implementation,a UART point-to-point communication is established between themicro-controller and the computer using a serial USB cable. Pressuredata is sent from the micro-controller to software drivers located on ahost computer.

In other embodiments, as seen in FIG. 39, there can be more than onemaster tile 7 in the grid. For larger areas and/or longer distances,groups of tiles can be reduced into zones, splitting up the dataresponsibilities to multiple masters 7. The data from these multiplezones can be collected through multiple communication protocols to thecomputer or a tree structure could be used so data is sent to a up thetree's masters until the data reaches the desired location. In otherembodiments, a multi-master protocol can be used to allow slaves 11 todivide the data sent between multiple masters in the same bus, reducingthe load on a single master 7 to collect the data. These masters can bebut are not necessarily the Host Communicator Tile 12 that transmitsdata to the computer.

Stepping through the entire process from the perspective of therespective parts of one embodiment:

List of Hardware Components

-   -   Host Computer 3    -   USB Connector 9    -   Printed Circuit Board 8    -   Microcontroller 5    -   Semi-Rigid Touch Layer 31    -   Active Sensing Array 20    -   Physical Substrate Support Surface 32    -   Inter-Tile Communication Cable 10    -   Neighbor Query/Sense Wires 13    -   Inter-Tile Physical Link Connector 71    -   Apparatus Housing/Frame 14

A computer 3 is connected to a grid of tiles 7 & 11 with a USB Connector9 to a Host Communication Tile 12 in a grid of Tiles as seen in FIG. 36.

An Inter-Tile Physical Link Connector 71 physically connects the tilesto each other, as seen in FIGS. 41, 42A, and 42B.

The Inter-Tile Physical Link connection 71 should be sized to maintainthe same distance between the adjacent tile's sensing elements and thestandard (in Tile) sensing element spacing.

FIG. 45 shows two adjacent tiles preserving inter-sensing element 26distance is preserved across tiles 2.

An Inter-Tile Communication Cables 10 connects tiles, in oneimplementation, in a daisy chain manner as seen in FIG. 38.

FIG. 38 shows a Chain of Slave Tiles 2 to the Master 7/HostCommunication Tile 12, and then via USB 9 to Computer 3.

The tiles do not need to be in any particular geometric configuration.In fact, the surface they form can be non-contiguous. FIG. 43 shows adaisy chain connection between an arrangement of non-contiguous tiles 2.The tiles 2 are connected by a daisy chain of inter tile connections 10.One of the tiles acts as master 7 and host connectivity tile 12 and hasa connection 9 to the host computer 3.

A Query/Sense wire (QSW) 84-87 also is connected between adjacent tiles.

-   -   The North QSW 84 will be connected to the South QSW 85 of the        tile above it (if it exists).    -   The South QSW 85 will be connected to the North QSW 84 of the        tile below it (if it exists).    -   The East QSW 86 will be connected to the West QSW 87 of the tile        to its left (if it exists).    -   The West QSW 87 will be connected to the East QSW 86 of the tile        to its right (if it exists).

FIG. 40 shows a Sample Grid of Tiles with N/S/E/W neighbor query/senseconnection.

In One embodiment as seen in FIG. 119, each Tile 2 consists of:

-   -   A Support Layer 32    -   A Printed Circuit Board (PCB) with a microprocessor 4        -   The Printed Circuit board 4 may be mounted on the bottom of            the Support Layer 32.        -   An Inter-Tile Communication Cable 10 is attached to the            Printed circuit board 4 for connection to an adjacent tile            2.        -   Four Query/Sense Connection Wires 84-87 are attached to the            Printed Circuit Board 4.        -   The Host Communication Tile Printed Circuit Board 95 for a            Host Communication Tile 12 will also have a USB connection            wire 9 for connecting with the Host Computer 3. In the case            of a single tile embodiment, that single tile's printed            circuit board 4 can also provide the functionality of the            Host Communication Tile.    -   An Active Sensing Array 20 consisting of an N×M grid of sensing        elements and control wires 23.        -   The active sensing array 20 is placed above the Support            Layer 32.        -   The active sensing array 20 is wrapped around the edge of            the Support Layer 32.        -   The active sensing array 20 is plugged into the tile PCB 4            using the connector Tails 25 on the Active sensing array 20.    -   Protrusions 30 are affixed on the outer face of the active        sensing array 20 at the corresponding sensing element 26        locations as in an active sensing array with attached        protrusions 55 embodiment is shown in FIG. 119.    -   A Semi-Rigid Touch Layer 31        -   The Semi-Rigid Touch Layer 31 is placed on top of the active            sensing array 20.

In one embodiment, the Active Surface Array 20, as seen in FIGS. 1-6,for an N×M grid of sensing elements consisting of:

-   -   One layer with conductor lines 23 for N rows    -   One layer with conductor line 23 for M columns    -   Force Sensitive Resistor (FSR) material 24 at the row/column        intersections    -   Connector Tail 25 with N and M wires corresponding to rows and        columns conductor lines respectively. The connector tails are        separated into banks of 16.

FIG. 119 shows Connector Tails 25 separated into banks of 16.

FIG. 46 is a block diagram of the electronics for a tile functioning asboth the Host communication Tile 12 and as a Master Tile 7. The hostcomputer 3 is connected host communication tile 12 via a standardprotocol such as USB where the data is transferred back and forth viethe Rx 78 and Tx 79 line. Power can be supplied, via the USB cable, fromthe computer 3, through a voltage regulator 76 as required by themicrocontroller 5. The active sensing array 20 is connected to thePrinted Circuit Board 4 by plugging the connector tails 25 of the activesensor array 20 into the tail connector clip 16 on the printed circuitboard 4. The Master Tile 7 communicates with slave tiles 11 via acommunication protocol such as I²C connected by inter-tile communicationcables 10. Power, or Vdd 98, is supplied to all slave devices fromeither the Master Tile 7 or via an external power supply 17 as needed.Adding a common ground to all active electronics, Vss 99, completes thecircuit.

FIG. 47 shows a block diagram for a slave tile 11. The Microcontroller 5is on the same power (Vdd 98)/ground (Vss 99) circuit as the othertiles, including the master tile 7. The active sensing array 20 isconnected to the PCB 4 by plugging the active sensing array 20'sconnector tails 25 into the connector tail clip 16 on the printedcircuit board 4. A slave tile 11 communicates with other tiles via acommunication protocol such as I²C connected by inter-tile communicationcables 10.

Tile Housing/Frame

The entire Tile 2 assembly may be housed in frame made of plastic orother materials.

The width of any housing frame perimeter must be thin enough to maintaininter-sensing element distances across tiles, as seen in FIG. 45.

Stepping through one embodiment of capturing and transmitting PressureImage Data across multiple tiles and to a Host Computer, to create afull time-varying multi-tile Pressure Image.

Each Tile contains (along with supporting electronics as per thedescription above):

-   -   A programmable microcontroller 5    -   Microcode to sensor data collection and communication (described        as follows)    -   An Active Sensing Array with N columns and M Rows 20    -   Inter Tile Communication wiring 10 to support a Master/Slave        bus, such as I²C, as shown in FIG. 38

The Host Communication Tile 12 (such as T-0 in FIG. 38) contains:

-   -   A USB Connection 9 to the Host Computer 3

Note: It is standard that commercial microprocessors provide intercircuit communication protocols such as I²C capabilities.

-   -   For example PIC24HJ256GP610 microcontroller from MicroChip        provides I²C support    -   I²C is an industry standard Master/Slave Bus Protocol    -   I²C provides protocols for dynamically assigning unique IDs to        slaves on the Bus

Note: It is standard that commercial microprocessors provide USBcapabilities

-   -   For example, PIC24HJ256GP610 microcontroller from MicroChip        provides USB support

Note: It is standard that commercial microprocessors can simultaneouslysupport both I²C and USB communications

-   -   For example PIC24HJ256GP610 microcontroller from MicroChip has        this capability

As per the above, the methodology will assume that

-   -   The Host Communication Tile 12 will contain Host Communication        Tile Firmware    -   In the example shown in FIG. 38, Tile T-0 is acting as the Host        Communication Tile 12 and as a Master Tile 7 for the grid    -   All Other tiles will be considered slave tiles 11    -   Slave tiles 11 will contain the slave tile Firmware    -   Slave tiles 11 will have obtained unique IDs as per I²C standard        protocol

Firmware on the microcontroller for tiles perform several distinct tasks

-   -   1. Local Tile Sensor Grid Pressure Image Capturing    -   2. Getting the Data from Slaves 11 to the Master Tile 7 and/or        Host Communication Tile 12    -   3. Communicating Local Tile Sensor Grid Pressure Image to Host        Computer 3    -   4. Communicating Tile topology and/or adjacency data to the Host        Computer 3 for the reconstruction of the multi-Tile Pressure        Image on the Host Computer 3    -   5. Initial Dynamic Discovery of neighboring tile topology        adjacency data        -   Note this step would not be necessary if pre-assigned IDs            were applied to the tiles along with manual storing of tile            topology.

In a single tile apparatus embodiment, that single tile can also acts asthe Host Communication Tile 12. In a single Zone apparatus embodiment,namely an apparatus containing grid of tiles with a single Master Tile 7and as seen in FIG. 38, that single Master Tile 7 can also act as theHost Communication Tile 12. In a multi-zone apparatus embodiment, namelyan apparatus containing grid of tiles with multiple Master Tile 7 incommunication with each other and as seen in FIG. 39, one of thesemaster tiles 7 can also act as the Host Communication Tile 12.

In some embodiments, the circuitry and microcode for the Master tilefunctionality may be on a separate printed circuit board that may or maynot physically be connected to the Master Tile 7. Similarly, in eachcase, in some embodiments, the circuitry and microcode for the HostCommunication Tile functionality may be on a separate printed circuitboard that may or may not physically be connected to the HostCommunication Tile 7.

Each connecting cable that goes between two tiles such as the Inter TileCommunication Cable 10 or the Master-master multi-zone connector cable94 is concurrently an ‘inbound cable’ for one of the tiles and ‘outboundcable’ for the other. Relative to a specific tile though, an ‘inboundcable’ is one from the tile in the chain from which sensing data packetsflow towards the host computer in the visa-versa for an ‘inbound cable’.For example relative to FIG. 38, the cable between T-1 and T-2 is anInbound cable for T-2 and an outbound cable for T-1.

FIG. 44 shows the cables/wires to/from a respective tile printed circuitboard 4 for one embodiment of tiles such that:

-   -   All tiles have Query Sensing Wires 84-87;    -   All Tiles have Connector Tails 25 going into their Connector        Tail Clip 16    -   Master Tile 7 and Non-Terminal slave tiles 11 for a zone have        Outbound Inter-Tile Communication Cables 89    -   Slave Tiles 11 have Inbound Inter-Tile Communication Cables 88    -   Host Communication Tile 12 will have a USB Cable (in one        embodiment)    -   In a multi-zone apparatus, Host Communication Tile 12 and        Non-Terminal Master tiles 7 for a zone have Outbound        Master-master multi-zone communication cable 74    -   In a multi-zone apparatus, Non Host communication Master Tiles 7        for a zone have Inbound Master-master multi-zone communication        cable 73

(1) Local Tile Sensor Grid Pressure Image Capturing (Both Master andSlave)

The Image Capturing Microcode will maintain N×M numeric Pressure ImageBuffer of measured sensing element values corresponding to a Frame ofpressure data for that tile. The values in this Buffer are measured inthe following manner.

-   -   The (i,j) element of the Pressure Image Buffer will correspond        to the pressure value for a row and column intersection.    -   As per method described in the text above, the (i,j) element of        the Image Buffer Array may be measured by        -   Setting all output wires to ground, except for the i-th            output wire        -   Set the i-th output wire to Positive        -   Set all input wires to ground, except for the j-th input            wire        -   The firmware will scan the j-th input wire reading it as a            digital value        -   This value will be stored in the (i, j) element of the            Pressure Image Buffer    -   By looping through all N and M wires a complete N×M Pressure        Image Buffer data is measured

(2) Getting the Data from Slaves Tiles 11 to the Master 7

The Microcode on the Master Tile 7 will poll each slave tile 11 forPressure Image Data

-   -   The reported data packet from each slave will contain the tile        ID and the Pressure Image Buffer Data    -   For simplicity, assume the Pressure Image Buffer Data is a full        copy of the Tile's Image Buffer        -   Alternatively it could be run length encoded        -   Alternatively it could provide delta (only changes from the            previously reported buffer)        -   Either, both or other techniques can be applied to improve            performance on the data transfer subsystem

The Microcode on the Slave Tiles 11 will receive a poll request andrespond by sending the packet of data as per the above description,namely Tile ID+Pressure Image Buffer data

(3) Communicating Local Tile Sensor Grid Pressure Image from the MasterTile 7 to Host Computer 3, described for the embodiment where the MasterTile 7 is also acting as the Host communication Tile 12.

Expanding upon (2) above, the Master Host Communication Tile 7 will

-   -   For Each Slave Tile 11        -   Poll each Slave Tile 11 for Pressure Image Data over the I²C            Bus        -   Receive the Slave Tile's 11 Pressure Image Data over the I²C            Bus        -   Send the Slave Tile's 11 Pressure Image Data to the Host            Computer 3 over USB    -   Send its own Pressure Image Data (if connected to a tile) to the        Host Computer 3 over USB

By repeating the above step continuously, Streaming, Time-VaryingPressure Image Data for the aggregate of tiles 2 will be received by thehost computer 3.

(4) Reconstruction a multi-Tile Pressure Image on the Host Computer

In one embodiment an A×B row/column grid of Pressure Tiles 2, eachcontaining N×M row/column grid of sensing elements 26 in theirrespective Active Sensing Arrays 20, produces an effective PressureSurface of (A*N) rows and (B*M) columns grid of addressable Pressuredata of a reconstructable pressure image.

A Tile Topology Data Table on the host computer can be maintained withthe position of the Tile relative to the overall Grid of Tile Topology

-   -   In one embodiment this can be manually stored on the Host        Computer    -   In another embodiment it can be dynamically constructed from a        Tile Adjacency Table

Sample Tile Topology and Tile adjacency tables corresponding to theapparatus configuration seen in FIG. 38 appear earlier in this document.

As Pressure Image Buffer Data for each tile with a provided Tile ID isreceived

-   -   The Tile Row r, and Tile Column c, values may be looked up in        Tile Topology Table    -   The Tile Pressure Image Data can be mapped to the Coherent        (N×A)×(M×B) overall pressure Image by mapping the tile's sensing        element data for (i,j) to (r*N+i, c*M+j)

(5) Initial Dynamic Discovery Neighboring Tile Topology

During an initialization phase, the relative positions of all of thetiles could be obtained by the following series of data exchanges (overthe I²C Bus unless otherwise stated).

The Microcode on the Master Tile 7 performs as follows:

-   -   For Each Slave Tile 11 and for the master tile 7        -   For each of North, South, East, West            -   Send a data packet requesting that the tile turn on the                corresponding Query/Sensing wire (North 84, South 85,                East 86, or West 87) for that direction for the query                Tile ID                -   Packet Contents: Query Tile ID and the direction                    wire to turn on            -   Receive the Query/Sense response packet from the                appropriate Tile                -   ‘Packet Contents: Detected’, direction                    (North/South/East/West), Detected Tile ID, Query                    TileID (from detecting Tile)                -   Packet Contents; ‘Nothing Connected’, direction,                    Query TileID            -   Send the response packet to the 3 Computer over USB

The Microcode on the Slave designated to receive the ‘activate wire’request to turn on the Query/Sensing Wire

-   -   If that tile detects that no tile is connected in the designated        direction (possibly due to an end resistor)        -   Send a ‘Nothing Connected’ response packet to the Master        -   Packet Contents; ‘Nothing Connected’, direction, Query            TileID        -   Otherwise, turn ‘on’ the designated directional Query            Sensing Wire (North 84, South 85, East 86, or West 87)

The Microcode on the Slave that detects the ‘on’ Query Wire State fromits corresponding Query State Wire (North 84, South 85, East 86, or west87)

-   -   Send a ‘Detected’ and its Tile ID data packet to the Master    -   ‘Packet Contents: Detected’, direction (North/South/East/West),        Detected Tile ID, Query TileID (from detecting Tile)    -   Note that the detecting wire direction is the opposite direction        as the detected tile direction, namely: detecting on North Wire        84 indicates tile to the South; South Wire 85 indicates tile to        the North; East Wire 86 indicates tile to the West; and West 87        Wire indicates tile to the East.

In the embodiment of an N×M Rectangular grid of tiles, a ‘Tile TopologyTable’ can be constructed from the ‘Tile Adjacency Table’ as follows:

-   -   Create a set of M ordered column lists of tile IDs corresponding        to North/South Connectivity by        -   for each of the M Tile IDs that has ‘none’ as its northern            neighbor            -   Search for the Tile ID that has this for its southern                neighbor            -   Iterate until a Tile ID with ‘none’ as its southern                neighbor is obtained    -   Order the set of M ordered Column lists left to right as        follows:        -   Search the set of Column Lists' first element for the one            with ‘none’ in the WEST direction. This is the leftmost            column (i.e. column 0)        -   Search for the Column List whose first Element is EAST of            the one just found        -   Iterate until at the column list who's first Element has no            EAST neighbor    -   One can now populate the Adjacency table by getting the        respective row/column numbers of the tile IDs        -   The column numbers are from the ordered column list position        -   The row numbers are the position in the respective column            list

A Description of the Actual Prototype that was Built

A Description as an Example of the Prototype Built: (a) the ActualMaterials Used for Each Layer, (b) the Dimensions, (c) the Size of EachTile, (d) how Many Tiles were Used, (e) the Product Number and Companywhich Made a Given Component.

Basically all Details about the Prototype. It can be in any Form, Suchas a Table or List, Whatever is Easiest to Provide the Information intothe Application.

(a) The Actual Materials Used for Each Layer

The individual sensing materials used for each sensing tile consists ofa 5 mil thick plastic substrate, printed silver electrodes (placed at ⅜″spacing) and small rectangles of FSR materials in the vicinity of thegrid intersections.

(b) The Dimensions

The active sensing area of each sensing tile is 12″×12″

(c) The Size of Each Tile

Each tile is 12″×12″ with ⅜″ spacing between wires.

(d) The Product Number and Company which Made a Given Component:

Component Table

Name Component Value Manuf Manuf Part No Distrib Distrib Part No QtyC1-C10 .1 uf capacitor .1 uF c0603c104k5ractu Mouser 80-c0603c10455r 10C11 10 uf capacitor 10 uF C0805C106Z8VACTU Mouser 80- 1 C0805C106Z8VCONN8 Molex 1 mm 16pin n/a Molex 52271-1679 Mouser 538-52271-1679 1 botziff connector CONN9 Molex 1 mm 16pin n/a Molex 52271-1679 Mouser538-52271-1679 1 bot ziff connector CONN10 Molex 1 mm 16pin n/a Molex52207-1685 Mouser 538-52207-1685 1 top ziff connector CONN11 Molex 1 mm16pin n/a Molex 52207-1685 Mouser 538-52207-1685 1 top ziff connector LBLED BLUE n/a Avago HSMN-C170 Mouser 630-HSMN- 1 Technologies C170 LG LEDGREEN n/a Avago HSMM-C170 Mouser 630-HSMM- 1 Technologies C170 LR LEDRED n/a Avago HSMC-C170 Mouser 630-HSMC- 1 Technologies C170 R1 R 100CRCW0603100RFKEA Mouser 71-CRCW0603- 1 Ohms 100-E3 R2 R 4.7KCRCW06034K70FKEA Mouser 71-CRCW0603- 1 Ohms 4.7k-e3 RLB R SMT 3.3K 3.3KVishay CRCW06033K30JNEA Mouser 71-CRCW0603J- 1 Ohms 3.3K-E3 RLG R SMT3.3K 3.3K Vishay CRCW06033K30JNEA Mouser 71-CRCW0603J- 1 Ohms 3.3K-E3RLR R SMT 3.3K 3.3K Vishay CRCW06033K30JNEA Mouser 71-CRCW0603J- 1 Ohms3.3K-E3 U1 PIC24HJ256GP610 n/a Microchip PIC24HJ256GP610- Mouser 579- 1I/PF 24HJ256GP610- P/PF U2 REG1117A 3.3 v Fairchild REG1117A-ND DigiKeyREG1117A-ND 1 USB USB-RS422 n/a FTDI, ltd. TTL-232R-3.3V-WE Mouser895-TTL-232R- 1 Transceiver 5V-WE Sensor Sensing Layers n/a ParlexVIP294 Parlex VIP294 2 Total Number of 27 parts:

(e) Pressure Sensitivity

To test the pressure sensitivity of the prototype, a 5 g base that restsfour points was placed with one of the points on top of a wireintersection. 5 g and 100 g weights were placed on the base to createweights from 5 g to 300 g. The intersection received a quarter of thisweight, so the weight range varied from 1.25 g to 75 g at theintersection. Values were only registered by the computer for weightsabove 2.5 g. The values from the computer scaled linearly from 46.87 to1320.71.

Weight at Intersection Value on Visualizer  0 g 0 2.5 g  46.87  5 g101.65 7.5 g  167.97 10 g 218.75 12.5 g  265.62 25 g 468.75 50 g 871.3475 g 1320.71

Outline

-   -   List of all components        -   Integrated Protrusion and Base Layer 42        -   Active Sensing Array 20        -   Semi-Rigid Touch Layer 33        -   USB Cable 9 and USB Transceiver 80        -   Computer 3        -   Master Tile 7    -   Operation: Outside point of view        -   One or more objects are placed into contact with the            Pressure Sensing Apparatus 1. The Pressure Sensing Apparatus            1 sends to the computer a two-dimensional array of pressures            corresponding to the space-varying pressure of the objects            upon the surface.        -   User touches the Pressure Sensing Apparatus 1 at multiple            locations, and the device indicates both location and            pressure at each location.

The Embodiment that follows is similar to the Semi-Rigid Touch Layerwith Protrusions 33 and the Active Sensing Array with attachedprotrusions 55 embodiments described above in all aspects other than howforce is transmitted to the sensing elements 26 on the Active SensingArray 20. In the Integrated Protrusion and Base Layer 42 assembly, thisis accomplished by an assembly where the Active Sensing Array 20 sitsbetween a Semi-Rigid Touch Layer 31 and an Integrated Protrusion andBase Layer 42 as seen in FIG. 19. All approaches result in imposition offorce 34 values being measured at each sensing element 26 on the ActiveSensing Array 20. As a result, the description of Interpolation,scanning of data from the sensing elements 26 by the Microcontroller 5,networking slave tiles 11 and master tiles 7, and all other techniquesbeyond the measuring of the sensing element 26 pressure are all areachieved in a similar manner.

The Integrated Protrusion and Base Layer 42 embodiment is potentiallyeasier and less expensive to manufacture and assemble than theSemi-Rigid Touch Layer with Protrusions 33. In this embodiment, theSemi-Rigid Touch Layer 31 can be independent of any individual pressuretile 2 and may seamlessly span an arbitrary number of pressure tiles 2.This makes assembly and alignment of the Pressure Sensing Apparatus 1significantly easier. Having a seamless Semi-Rigid Touch Layer 31 alongadjacent pressure tiles 2 naturally results in identical and seamlessdistribution of force to sensing elements 26 regardless of whether thesensing elements 26 are on the same or adjacent pressure tiles 2.

Additionally, an embodiment of the Integrated Protrusion and Base 42Layer may includes housing for the Printed Circuit Board 4 and groovesfor Tile Connection Cables such as the Inter-Tile communicationConnection Cables 10 and multi-zone cable 94, thus reducing the numberof parts in the Pressure Tile 2 assembly.

The Pressure Sensing Apparatus 1 can incorporate a mechanical forceredistribution mechanism that properly distributes pressure so that thesensed pressure is well distributed to the sensing elements in the tile.

The Semi-Rigid Touch Layer with Protrusions 30 can be replaced by acomponent that is mechanically integral to the supporting base of thepressure tile 2 itself. This makes manufacture easier, is less expensiveand more robust, and that makes it easier to avoid misalignment betweensensing elements 26 and protrusions 30.

In order to create a Pressure Sensing Apparatus 1 of multiple pressuretiles 2 that creates a seamless and continuous interpolating touchresponse, the only mechanical component that needs to be shared betweenthe plurality of pressure tiles 2 is a featureless sheet of material,such as plastic, the position of which does not need to be preciselyregistered with the positions of the sensors in the grid of sensortiles.

Step by step description of internal working:

Internal operation begins when fingers or other objects impose downwardforce 34 upon the Semi-Rigid Touch Layer 31.

This force is then transmitted, and properly redistributed, from theSemi-Rigid Touch Layer 31 through the sensing elements 26 in the ActiveSensing Array 20. The force impinging on each sensing element 26 is thenimparted onto the corresponding protrusion 30 in the IntegratedProtrusion and Base Layer 42. This creates a concentration of force onthe portion of the Active Sensing Array 20 where each sensing element 26is in contact with a corresponding protrusion 30, thereby creating aforce that compresses together the two areas of FSR material 24 inmutual contact at the regions of the Active Sensing Array 20 thatcomprise the sensing elements 26 (where one FSR region on the outerconducting line of the Active Sensing Array 20 is in contact with acorresponding region of FSR material 24 on the Conductive Trace Lines 23of the Active Sensing Array 20).

The compression creates an increase of electrical conductance betweenthose two areas of FSR material 24 in mutual contact. As the sensor'smicro-controller 5 scans through the Active Sensing Array's 20 matrix ofsensing elements 26, each change in conductance is measured as a changein voltage, which the micro-controller detects via an Analog to DigitalConverter (ADC) 83 that the microcontroller 5 then encodes as a digitalsignal. The microcontroller 5 then sends this digital signal through aUSB Cable 9 to a host computer 3.

Unlike the Semi Rigid Touch Layer with Protrusions 33 technique wherethe inner face of the protrusions 30 are in contact with the outersurface of the Active Sensing Array 20 as seen in FIG. 15, thistechnique with the Integrated Protrusion and Base Layer 42 has the outerface of the protrusions 30 in contact with the inner surface of theActive Sensing Array 20, as seen in FIG. 121. This mechanicalarrangement allows a concentration of force at the sensing elements 26of the Active Sensing Array 20, thereby enabling spatial interpolationbetween adjoining sensing elements 26 without the requirement ofprotrusions 30 above the Active Sensing Array 20.

One microcontroller 5 can be associated with each pressure tile 2.

General Purpose of Each Layer

FIG. 19 shows an exploded view of a Pressure Tile 2 with the followingcomponents: Integrated Protrusion and Base Layer 42, 2 Active SensingArray 20, Semi-Rigid Touch Layer 31. The Conductive Trace Line 23intersections on the Active Sensing Array 20 are the locations of theFSR material 24 sensing elements 26. When the layers are placed intocontact, each intersection in the Active Sensing Array 20 is alignedwith the center of a corresponding protrusion 30 in the IntegratedProtrusion and Base Layer 42.

FIG. 20 shows a Profile view of a Pressure Tile 2 with: the Semi-RigidTouch Layer 31 which is in contact with the Active Sensing Array 20 2;and the Active Sensing Array 20 which is in contact with protrusions 30of the Integrated Protrusion and Base Layer 42. The protrusions 30 onthe Integrated Protrusion and Base layer 42 are aligned with the sensingelement 26 regions on the Active Sensing Array 20.

Active Sensing Array 20: The Active Sensing Array 20, shown in FIG. 1,consists of two sensor surface sheets 21 facing each other, where onesensor surface sheet 21 is rotated 90° with respect to the other sensorsurface sheet 21, as seen in FIG. 2. FIG. 4 shows the layers of a SensorSurface Sheet 21 that is complete in FIG. 3. Upon each of the two sensorsurface sheets 21 is printed conductive trace lines 23. Small amounts offorce sensitive resistive (FSR) material 24 is printed at intervals suchthat when the two substrates are placed into mutual contact, with theFSR material 24 sides facing each other, the FSR material 24 printed oneach sensor surface sheet 21 is place in the vicinity of theintersections of the grid of conductive trace lines 23. The gridintersection points of overlapping FSR material 24 comprise a sensingelement 26 where pressure may be measured.

The Integrated Protrusion and Base Layer 42 consisting of a grid ofprotrusions 30 spaced such that when the Active Sensing Array 20 isaffixed over this layer, one of these protrusions 30 sits directly undera sensing element 26 of the Active Sensing array 20 at the junctions ofa multitude of row and column electrodes where the FSR material 24layers are sandwiched so that pressure may be measured at eachintersection point.

The Semi-Rigid Touch Layer 31 is placed in contact with one or moreActive Sensing Arrays 20, each of which is resting in contact with theprotrusions 30 in its respective Integrated Protrusion and Base Layer42. Pressure applied to the Semi-Rigid Touch Layer 31 will focus theforce to the sensing elements 26 directly above protrusions 30 on theIntegrated Protrusion and Base Layer 42. In one implementation, theSemi-Rigid Touch Layer 31 is implemented as a sheet of vinyl that can bein the range of 0.5 mm to 1.0 mm in thickness. In another implementationof a single tile configuration the Non-Conductive Surface Substrate 22of the Active Sensing Array 20 may act as its own Semi-Rigid Touch Layer31. In other implementations the Semi-Rigid Touch Layer 31 may be madeof glass, metal or any other material whose thickness can be chosen sothat the Semi-Rigid Touch Layer's 31 rigidity falls within a usefulrange of rigidity.

FIGS. 21, 22 and 23 are three cases in which the Semi-Rigid Touch Layer31 is, respectively: FIG. 21 Too rigid; FIG. 22 Within the useful rangeof rigidity; FIG. 23 Insufficiently rigid. In each case, the hand showsImposition of Force 34, and the arrows show imparted force transmittedto the base 56 to different parts of the base 32 of the pressure tile 2.

The Semi-Rigid Touch Layer 31 having a “useful range of rigidity” can bedefined via the following constraints of maximal rigidity and minimalrigidity, respectively: The Semi-Rigid Touch Layer 31 would be too rigidif an externally applied force within a 1 mm diameter circular region ofthe outer face of the Semi-Rigid Touch Layer 31, lying within arectangular region bounded by four nearest protrusions 30 of theIntegrated Protrusion and Base Layer 42 at the rectangle's corners, wereto result in pressure being applied to protrusions 30 of the IntegratedProtrusion and Base Layer 42 other than those four nearest protrusions30, as shown in FIG. 21. For example, a 1 cm thick plate of glass wouldbe too rigid to serve as the Semi Rigid Touch Layer 31. 2 The Semi-RigidTouch layer 31 is in the useful range of rigidity if the imposion offorce 34 causes force to be imparted to those nearest protrusions 30 butnot to other protrusions 30 of the Integrated Protrusion and Base Layer42, nor to the underlying surface of the Support Layer 32 between theprotrusions 30 as shown in FIG. 22; The Semi-Rigid Touch Layer 31 wouldbe insufficiently rigid if the same imposion of force 34 were to causethe Semi-Rigid Touch Layer 31 to deform to sufficient extent that theSemi-Rigid Touch Layer 31 would physically come into contact with theregion of the Integrated Protrusion and Base Layer 42 between those fourprotrusions 30, thereby dissipating force onto inactive regions of theActive Sensing Layer 20 as shown in FIG. 23. For example, a 0.5 mm thicksheet of rubber would be insufficiently rigid to serve as the Semi-RigidTouch Layer 31.

In one implementation the Semi-Rigid Touch Layer 31 consists of a 1.0 mmthick sheet of vinyl which has a Young's Modulus of elasticity ofapproximately 0.33 GPa's or 49000 psi would fall into the valid range ofrigidity for the prototype implementation with ⅜″ spacing of protrusionsthat are 1 mm in height. Other materials would suffice, but as theYoung's Modulus increases, the thickness of the material shouldcorrespondingly decrease so as to localize the bending or elasticity ofthe material to a region of no more than 2×2 square sensing elements 30.

The total size and shape of the Semi-Rigid Touch Layer 31 can be made soas to match the total size and shape of the networked grid of pressuretiles 2 in the apparatus 1.

An Integrated Protrusion and Base Layer 42 contains a grid of upwardfacing protrusions 30 spaced such that when the Active Sensing Array 20is placed on the outside face of this layer, each of these protrusions30 is aligned with active sensing area 27 of one of the sensing elements26 of the Active Sensing Array 20, as seen in FIG. 20.

A Semi Rigid Touch Layer 31 is placed in contact on the outside face ofthe Active Sensing Array 20. Imposition of force 34 applied from aboveto this Touch Layer will be focused by the geometric arrangement ofsensing elements 26 that are in contact with corresponding protrusionsof the Integrated Protrusion and Base Layer 42 so that all appliedpressure 34 imparted to the Semi-Rigid Touch Layer 31 becomesconcentrated in the region where the sensing elements 26 of the ActiveSensing Array 20 are in contact with corresponding protrusions 30 of theIntegrated Protrusion and Base Layer 42, as seen in FIG. 20.

This configuration of components forms a mechanism whereby a continuouschange in position of a touch on the outer face of the Semi-Rigid TouchLayer 31 results in a corresponding continuous change in the relativeforce applied to the active areas 27 of those sensing elements 26 thatare nearest to that touch, as shown in FIG. 24. Those relative forces,when sent to the host computer 3 as part of the data image, permit thehost computer 3 to accurately reconstruct the centroid position of thetouch through arithmetic interpolation.

FIG. 24 shows a three dimensional view of interpolation: The impositionof force 34 impinging upon the Semi-Rigid Touch Layer 31 at a givenlocation will be focused on the 2×2 nearest protrusions 30 of theIntegrated Protrusion and Base Layer 42. Therefore in the Active SensingArray 20 layer all of the imposition of force 34 will be concentrated onthe 2×2 active sensing areas 27 of the sensing elements that are indirect mechanical contact with these four protrusions 30.

Functional Layers

The three components of, respectively, the Semi-Rigid Touch Layer 31,the Active Sensing Array 20, and the Integrated Protrusion and BaseLayer 42, can be seen as consisting of five functional layers, for thepurposes of describing the internal mechanism of operation at a singlesensing element as seen in FIG. 121.

These functional layers are, respectively:

(1) the Semi-Rigid Touch Layer 31;

(2) the Active Sensing Array 20 consisting of: outer Non-conductivesurface substrate 22, outer Conductive trace lines 23 (not shown in thisFIG. 121); inner and outer FSR material 24 layers; inner Conductivetrace lines 23 (not shown in this FIG. 121); and inner Non-conductivesurface substrate 22; and

(3) the Integrated Protrusion and Base Layer 42 containing protrusions30.

The semi-rigid Touch Layer 31 redistributes the applied forced 34 suchthat all force 34 is distributed only to the sensing elements 26 in theActive Sensing Array 20. The focusing is accomplished at the contactpoints at the protrusion 30 on the Integrated Protrusion and Base Layer42 and the active sensing area 27 corresponding to a sensing element 26on the active sensing array 20, as seen in FIG. 20.

In one embodiment, the outer non conductive surface substrate 22 of thesensor surface 21 of the Active Sensing Array 20, which can be made ofthin acetate which can, in one implementation, be 5 mils in thickness,together with the conductive trace lines 23 which are printed on theinner face of the non-conductive surface substrate 22. FSR material 24is printed over the conducting lines of the inner face of the outersurface sheet 21 of the Active Sensing Array 20 and the conducting linesof the outer face of the inner sensor surface sheet 21 of the ActiveSensing Array 20. In operation, these two FSR material 24 components arein contact with each other, but are not mechanically affixed to eachother. The inner non conductive surface substrate 22 of the inner sensorsurface sheet 21 of the Active Sensing Array 20, which can be made ofthin acetate which can, is, in one implementation, 5 mils in thickness,together with the conductive trace lines 23 which is printed on theouter face of their non-conductive surface substrate 22.

The Integrated Protrusion and Base Layer 42 contain the protrusions 30.Its purpose as the base of the pressure tile 2 is to provide theprotrusions 30 so that the force applied to the Semi-Rigid Touch Layer31 only to the active area of the corresponding sensing element 27 onthe Active Sensing Array 20.

Interpolation Involving a Plurality of Pressure Tiles 2

With a networked tile assembly 18 of adjacent pressure tiles 2, theSemi-Rigid Touch Layer 31 can consist of a single uninterrupted sheet ofsemi-rigid material (such as thin semi-rigid plastic), which covers allof the pressure tiles 2 in the grid of pressure tiles 2. This has theadvantage that the mechanical interpolation process of neighboringsensing elements 30 in the Active Sensing Array 20 Layer of differentadjoining pressure tiles 2 is identical with the mechanicalinterpolation process of neighboring sensing elements 30 within eachindividual pressure tile 2. The effect from the user's perspective is aninterpolating touch response that is exactly equivalent to theinterpolating touch response of a single extremely large pressure tile2.

Note that in this arrangement, there is no need for exact registrationbetween the Semi-Rigid Touch Layer 31 and the individual pressure tiles2, since the Semi-Rigid Touch Layer 31 itself can be a featureless anduniform sheet of material.

The nearby protrusions 30 and corresponding sensing elements 26 do notneed to be on the same pressure tile 2, but rather can be on adjacent,mechanically separate, tiles, as in FIG. 122.

In one implementation, as seen in FIG. 122, the semi-rigid Touch Layer31 spans the totality of pressure tiles 2. Pressure applied in a regionbetween two pressure tiles 2 transmit force to the nearby supportingprotrusions 30 on two adjacent but mechanically distinct pressure tiles,and thence to sensing elements 30 of the Active Sensing Arrays 20 withintwo distinct pressure tiles.

When pressure tiles are adjacent, such as a in a Network Tile Assembly18, the Semi-Ridged Touch Layer 31 will span the totality of thesurface, overlapping all the spaces between the underlying pressuretiles 2. As long as adjacent pressure tiles 2 are properly registered sothat the distance between protrusions 30 on each pressure tile 2 ismaintained across adjacent pressure tiles 2, then the interpolatingforce distribution across adjacent sensor tiles will be identical tothat within a single pressure tile 2. In one embodiment, pressure tile 2registration can be accomplished by having alignment brackets on eachindividual sensor tile as seen in FIGS. 41, 41A, 42B.

Three Cases of Interpolation:

1) FIG. 25 shows a region 69 where force would be distributed to fourprotrusions 30 on the same pressure tile 2.

2) FIG. 26 shows a region 69 where force would be distributed to twoprotrusions 30 on each of two adjacent pressure tiles 2. Pressureapplied in a region on the edge where two pressure tiles 2 meettransmits force to the nearby supporting protrusions 30 on the twoadjacent but mechanically distinct pressure tiles 2 and thence topressure senses of the Active Sensing Arrays 20 of two pressure tiles 2.The uninterrupted Semi-Rigid Touch Layer 31 spans the two pressure tiles2. Pressure applied along the edge of the adjacent pressure tile 2 willdistribute the force to the four sensing elements 26 (two on eachrespective pressure tile 2) in the same manner as if those sensingelements 26 had been on the same tile. The interpolation methods canthen treat the pressure values across adjacent pressure tiles 2 as if itwere a coherent larger ‘image’.

3) FIG. 27 shows a region where force 69 would be distributed to oneprotrusion 30 on each of four adjacent pressure tiles 2. Pressureapplied in a region at the corner 125 where four pressure tiles 2 meettransmit force to the nearby supporting protrusions 30 on the fouradjacent but mechanically distinct pressure tiles 2 and thence topressure sensitive sensing elements 30 where conductive trace lines 23intersect on the Active Sensing Arrays 20 of four distinct pressuretiles 2, as seen in FIG. 27. The uninterrupted Semi-Rigid Touch Layer 31spans the four pressure tiles 2. Pressure applied at the corner 125 ofthese adjacent pressure tiles 2 will distribute the applied force tothose four sensing elements 26 (one on each respective sensor tile) inthe same manner as if the sensing elements 26 had been on the samepressure tile 2. The interpolation methods can then treat the pressurevalues across adjacent pressure tiles 2 as if it were a part of a singlecoherent larger ‘image’ of pressure.

The term ‘image of pressure’ is used here to denote a two dimensionalarray of pressure values. The image generated by the current inventionis antialiased, as per the commonly accepted definition of the term‘antialiased’, in that pressure imparted in any area-variant pattern tothe outside surface of the Semi-Rigid Touch Layer 31 is converted by theplurality of pressure tiles into a band-limited representation of theoriginal area-variant pressure pattern that is faithful to the originalpattern for all spatial frequencies lower than a upper boundingfrequency that is determined by the grid resolution of each tile'sActive Sensing Array 20.

The Integrated Protrusion and Base Layer 42 can be a single mechanicalcomponent, which can be made of plastic, glass, wood, metal, or anyother semi-rigid material. This component can be manufactured by avariety of standard methods, including injection molding, stamping, andcold casting.

In an alternate embodiment, a rapid prototype for the IntegratedProtrusion and Base Layer 42 may be manufactured via SLA methods. In onemethod of manufacture, a mold, which can consist of silicone rubber, maybe made from this prototype. Resin may be poured into the mold. When theresin hardens, the mold is removed, and the resin forms a functionalIntegrated Protrusion and Base Layer 42.

Advantages of protrusions 30 being underneath: Integrating theprotrusions 30 with the pressure tile 2 into a single mechanical partmakes it easier to register the positions of multiple pressure tiles 2.Registering the positions across pressure tiles 2 is important ineffecting an interpolation scheme that behaves the same across aplurality of pressure tiles 2 as it does within a single pressure tile2. By making the support layer 32 that contains the protrusions 30 anintegral part of the sensor tile, registering protrusions 30 acrosssensor tiles is accomplished by just mechanically attaching eachpressure tile 2 to its neighbors.

In one implementation, the Integrated Protrusion and Base Layer 42 wouldbe made of injection molded plastic or cast resin from a silicone rubbermold, and would consist of a 12″×12″ rectangular base with a grid of32×32 upward facing protrusions 30 with spacing between the centers ofthe protrusions of ⅜″ (corresponding to the inter-sensing elementspacing of Active Sensing Array 20) and the height of the protrusionswould be 2 mm.

In one implementation of the Integrated Protrusion and Base Layer 42, asseen in FIG. 33 and FIG. 34, the base would be molded with a cavity onits inner face, to house the pressure tile's 2 Printed Circuit Board 4,as shown in FIG. 33 and FIG. 34. Channels would also be molded into theIntegrated Protrusion and Base Layer 42 to support Tile ConnectionCables 17.

In another implementation, the Integrated Protrusion and Base Layer 42face opposite the protrusions 30 may be flat. This flat side may bemounted onto a separate support layer 32 such as a ¼″ thick sheet ofacrylic with a cavity cut on inner face to house the sensor tile'sPrinted Circuit Board 4. Channels would also be cut into the Base Layer32 to support Tile Connection Cables 10. In this implementation, theshape of the Integrated Protrusion and Base Layer 42 part would have aflat bottom as in FIG. 32, but laying upon a base layer 32 with thecavity in it.

If the pressure tile's 2 Printed Circuit Board 4 is located underneaththe device, then the Active Sensing Array 20 must be wrapped around theIntegrated Protrusion and Base Layer 42. When the Active Sensing Array20 is wrapped too tightly around the Integrated Protrusion and BaseLayer 42, then unwanted force will be applied to protrusions 30, andtherefore to sensing elements 26, near the edge of the IntegratedProtrusion and Base Layer 42. If the Active Sensing Array 20 is wrappedtoo loosely, then it can bow up and cause a loss of sensitivity at thosesensing elements 26. In order to prevent these situations, adhesive 40can be placed on both the protrusion 30 side and the Semi Rigid TouchLayer 31 side of the Active Sensing Array 20.

In one implementation of the Integrated Protrusion and Base Layer 42,which was made using standard rapid prototyping techniques, theprotrusions 30 are made of ABS plastic and are each 2 mm in height and 4mm wide at their base, with spacing between adjacent protrusion centersof ⅜″.

The height, shape and curvature at the peak of the protrusions 30 mayvary based upon the application of the pressure tile 2. The shape of theprotrusion 30 may affect the spread of force onto the active area of thesensing element 27 and durability of the apparatus.

In one implementation, as seen in FIG. 28 showing tall/narrowprotrusions, each protrusion 30 may be longer than it is wide, with arounded tip, such as a paraboloid shape with a diameter at its base of 4mm and a height of 4 mm. This configuration focuses the force into asmall area of the sensing element 26 with which the protrusion 30 is incontact, thereby giving the greatest sensitivity. Such a configurationis preferred for creating a pressure tile 2 that is sensitive to verylight pressures, but is less preferred for sharp or heavy touchesbecause high pressures may result in damage to the Active Sensing Array20.

In another implementation, as seen in FIG. 29, the protrusions 30 may behemispherical such as with a diameter at the base of 4 mm and a heightof 2 mm. This shape has the benefit of providing greater mechanicalstrength, while also keeping the curve at the top of the protrusion 30gradual thereby reducing the danger of mechanical damage to the ActiveSensing Array 20 during very high pressure loads.

In another implementation, as seen in FIG. 30, the protrusions 30 mayhave a paraboloid or sinusoidal shape that is much wider, such as aparaboloid with a diameter at its base of 4 mm and a height of 1 mm.This retains most of the advantages of a hemisphere shape whileproviding the benefit of being easier to fabricate using less expensivecasting methods than a hemispherical protrusion since a paraboloid orsimilar shape does not have a vertically intersecting wall at its base.

In another implementation, as seen in FIG. 31, the protrusions 30 may bevery wide, with a paraboloid or sinusoidal shape, such as a paraboloidwith a diameter at its base of 8 mm and a height of 2 mm. Thisconfiguration results in a very gradual curve at the top of theprotrusion 30, thereby minimizing the chance of damage to the sensorarray when sharp or heavy pressure is applied.

Single Tile Assembly 48

In one single tile assembly 48 embodiment, a single pressure tile 2 maybe directly connected to a computer and does not require a masterprinted circuit board 19, though a distinct or integrated Hostcommunication Printed Circuit 38 is needed. Such an embodiment isassembled, as seen in FIG. 32 and FIG. 33, with the flexible ActiveSensing Array 20 wrapped around the edge of the tile, and plugged intothe Tile Printed Circuit Board 4 which is affixed to the underside ofthe Integrated Protrusion and Base Layer 42. The Semi-Rigid Touch Layer31 sits on top of the Active Sensing Array 20. In the single tileembodiment the Microcontroller 5 on the Tile Printed Circuit Board 4 canperform both the scanning and Host Communication, such as USB via a USBcable 9 with the computer 3, as seen in FIG. 35.

Networked Tile Assembly 18 of a Plurality of Pressure Tiles 2:

In one multiple tile embodiment, slave tiles 12 may be daisy chained toa master tile 7 or master printed circuit board 19 which may haveintegrated or separate Host communication circuitry 95 which isconnected to a computer 3. Such an embodiment is assembled seen in FIG.50 with series of slave tiles 12 connected to a Master Printed CircuitBoard 19, allowing for a master/slave bus protocol for getting pressuredata from the series of slave tiles. The Semi-Rigid Touch Layer 31 spansthe slave tiles 11, on top of their respective individual Active SensingArrays 20. The Microcontroller 5 on the Master Printed Circuit Board 19gathers data from the slave tiles 11 and transmit that data to the HostCommunication Circuitry 95 which transmit the data via a USB transceiver30 via the USB cable 9 with the computer 3.

Pressure Sensitivity

To test the pressure sensitivity of two prototypes, a 5 g base whichrests on four points was placed on top of the semi-rigid touch layer 31with each point above a sensing element 26. 5 g weights were placed onthe base to create weights from 5 g to 100 g. Each sensing element 26received a quarter of this weight, so the mass varied from 1.25 g to 25g at each sense.

Test 1:

Touch Layer-0.5 mm Vinyl

Sensor-108 kOhm resistive ink sensor.

Protrusion layer-4 mm diameter hemispheres

Weight At Intersection Average Value on Visualizer  5 g 0 10 g 7.5 15 g14.5 20 g 23 25 g 32

In this implementation, masses below 10 g are not registered by thepressure tile 2. After 10 g, the average value registered by thepressure tile 2 scaled linearly with pressure.

Test 2:

Touch Layer-1 mm Vinyl

Sensor-108 kOhm resistive ink sensor.

Protrusion layer-2 mm diameter truncated cones

Weight At Intersection Average Value on Visualizer  5 g 0 10 g 0 15 g 220 g 17.5 25 g 25

This test used a thicker semi-rigid touch layer 31, which makes the toplayer more ridged but decreases the sensitivity. As a result, valueswere not registered until 15 g.

In this extension on the ideas of the above embodiments encompassing animproved technique for concentrating force to the appropriate sensingelements 26 on an Active Sensing Array 20. In this embodiment, the touchsurface lies over plates 35 spanning the sensing elements 26 such thatthe plate corners are aligned with the protrusions 30. This eliminatesthe range of rigidity requirements of the Semi-Rigid Touch Layer 31 inthe above embodiment, instead utilizing a Flexible Touch Layer 38 asthis touch layer lays flat on the plates 35 which in turn focus theforce onto the appropriate sensing elements 26. As a result the FlexibleTouch Layer 38 can be thin and flexible for example 1/10th the thicknessand rigidity as with the prior invention (e.g. a 5 mil sheet of PETfilm). Such a thin/flexible touch layer on top of plates eliminates theundesired spread along the touch layer of applied force beyond thesensing elements in the immediate proximity of that applied force.

Additionally, because the Flexible Touch Layer 38 lies flat on theplates 35, rather than on the protrusions 30, this embodiment allows theuser to interact with the device without feeling protrusions/bumps 30through the touch layer. Also because the Touch layer lies on a plateauof plates 35, rather than bridging protrusions 30 as in the priorinvention, the Flexible Touch layer 38 can be more tightly adhered tothe plates 35, reducing compression issues that otherwise arise. Thisresults in a lower initial detectable touch threshold, improvingdetection of light touches.

This technique provides a more efficient mechanism for transmittingforce from the touch layer to the sensing elements than the priorinvention because all dissipation of force is done on the microscopiclevel rather than the macroscopic level. The above embodiments requiredsome rigidity (described as in a ‘Useful Range of Rigidity’) on thetouch layer since the touch layer was used to spread force to thesensing elements via a macroscopic deformation of the touch layer. Inthis invention there is no macroscopic movement or deformation, onlymicroscopic deformations due to: deformation of the plate; compressionof the protrusions; or and/or hinging where the plates meet each otherand/or the protrusions. This results in reduced loss of pressure signaldue to deformation; a higher percentage of force goes to local sensingelement rather than being transmitted to further neighboring sensingelements.

The step by step description of the user experience is the same asdescribed above for this embodiment.

List of All Components

A List of all Hardware Components.

-   -   List of all components        -   A collection of sensor tiles 2, where            -   A sensor tile consists of:                -   Flexible Touch Layer 38                -   Adhesive Layer (s) 40                -   Technique: Integrated Plate and Protrusion Matrix                    Component                -    Integrated Plate and Protrusion Layer 36                -    Base Layer 47                -   Technique: Distinct Plate and Protrusion Matrix                    Components                -    Plate Layer 53                -    Integrated Protrusion and Base Layer 42                -   ALL OTHER COMPONENTS ARE AS DESCRIBED ABOVE        -   ALL OTHER COMPONENTS ARE AS DESCRIBED ABOVE

General Purpose of Each Layer: Integrate Plate and Protrusion LayerEmbodiment

FIG. 52 shows an exploded view of a Tile for the Integrated Plate andProtrusion Matrix Component embodiment: Flexible Touch Layer 38,Integrated Plate and Protrusion Layer (IPPL) 36, Active Sensing Array20, Base Layer 47. When the layers are placed into contact, eachprotrusion in the IPPL 36 is aligned to be in contact with the activearea of the sensing element 27 on the outside surface of the ActiveSensing Array 20. An Adhesive Layer 40 may also be used between theFlexible Touch Layer 38 and the IPPL 36 so these layers are mechanicallyconnected. Similarly, an Adhesive Layer 40 may also be used between theIPPL 36 and the Active Sensing Array 20. Similarly, an Adhesive Layer 40may also be used between the Active Sensing Array 20 and the Base Layer47.

This Integrated Plate and Protrusion Matrix Component embodiment of theinvention pertains to a pressure sensor which utilizes a differentmechanism for focusing force to the sensing elements in the activeSensing Array 20 than described earlier in this document. In thisembodiment as seen in exploded view and FIG. 52 in side view in FIG. 53,the Flexible Touch Layer 38 which is in contact with the IntegratedPlate and Protrusion Layer 36 which is in contact with the ActiveSensing Array 20; which is in contact with the Base Layer 47. Eachprotrusion 30 in the IPPL 36 is aligned to contact the correspondingactive area of a sensing element 27 on outside surface of the ActiveSensing Array 20, as seen in FIG. 52 and FIG. 53.

The Distinct Plate Matrix and Protrusion Matrix Layers embodiment ofthis invention pertains to another technique shown in exploded view inFIG. 54 and in side view in FIG. 55 in which there is a Flexible TouchLayer 38, Plate Matrix Layer 53 Active Sensing Array 20, IntegratedProtrusion and Base Layer 42. When the layers are placed into contact,each protrusion 30 in the Protrusion Layer 53 is aligned to contact thecorresponding active area of a sensing element 27 on inner surface ofthe Active Sensing Array 20. Additionally, the corners of each plate 35in the Plate Matrix Layer 53 are aligned with the correspondingprotrusions 30 from the Protrusion Layer 53 on the outer surface of theactive sensing array 20, where any protrusion may have up to fouradjacent plate corners above it.

An Adhesive Layer 40 may also be used between the Flexible Touch Layer38 and the Plate Matrix Layer 53 so these layers are mechanicallyconnected. Similarly, an Adhesive Layer 40 may also be used between thePlate Matrix Layer 53 and the Active Sensing Array 20. Similarly, anAdhesive Layer 40 may also be used between the Active Sensing Array 20and the Integrated Protrusion and Base Layer 42.

In an alternate embodiment, seen in FIG. 56, the protrusions are affixedto the active areas of sensing elements 27 on the outer surface of theActive Sensing Array 20. In this embodiment, the protrusions 30 and theActive Sensing Array 20 together form a single component of the device,the Active Sensing Array with attached Protrusions Layer 55. Inoperation, as seen in the exploded view in FIG. 57 the Flexible TouchLayer 38 rests atop the Plate Matrix Layer 53 which rests atop theActive Sensing Array with attached Protrusions Layer 55, which restsatop a base layer 47. When an external force is applied to the FlexibleTouch Layer 38, that force is then imparted to the Plates Matrix Layer53, which redistributes the force that that it becomes concentrated atthe corners of the plates 35, from which it is then imparted to theProtrusions 30, thereby compressing each active sensor 26 between theaffixed Protrusion 30 and the base layer 47 upon which the ActiveSensing Array 20 lies atop.

Glossary of Terms and Description of Components for this Embodiment

Active Sensing Array (ASA): Described Above

Sensing element 26: is at the location between the two Surface Sheets 21of the Active Sensing Array 20 where Conductive Trace Lines 23 cross,and at which two areas of FSR 24 are sandwiched together and thatpressure may be electrically measured, as seen in FIG. 10 and FIG. 11.The sensing element 26 is the area where there is an overlap of the FSRon those two layers at a junction of intersecting Trace Lines 23 as seenin FIGS. 9 and 10.

In Contact with a Sensing element: The Active Area of a Sensing element27 is the area on either side of the Active Sensing Array 20corresponding to the overlap of the FSR material for that sensingelement as seen in FIGS. 10 and 11. In particular, a protrusion 30 issaid to be in contact with a sensing element 26 if the surface of theprotrusion in contact with the Active Sensing Array 20 lies completelyupon or inside of the Active Area 27 of that sensing element. Aprotrusion 30 is properly aligned with the sensing element 26 if it isin contact with the sensing element (as just defined).

Plate 35: a rectangular piece of plastic, metal, wood, glass, or othersuch material that has a Valid Amount of Plate Rigidity (relative to theprotrusion heights, both defined below). The plate 35 is of a size andshape such that when it is positioned, the corners are aligned inside ofthe four adjacent sensing elements 26 on the Active Sensing Array 20.Plates 35 are arranged in a Plate Matrix 39 which may be a constituentof an Integrated Plate and Protrusion Layer (IPPL) 36 or part of a PlateMatrix Layer 53. FIG. 59 and shows a plate 35 properly aligned upon theActive Sensing Array 20. FIG. 60 shows the top view of Rigid Plate 35properly aligned, and inside of corresponding sensing elements 26 on theActive Sensing array 20.

Plate Matrix 39: A plurality of Rigid Plates 35 spatially aligned suchthat there is a gap between the Rigid Plates 35 and that the center ofthe gap between the corners is aligned to correspond with a sensingelement 26 on an Active Sensing Array 20. A Plate Matrix 39 may be aconstituent of an Integrated Plate and Protrusion Layer (IPPL) 36 or ofa Plate Matrix Layer 53. FIG. 61A shows the top view and FIG. 61B theside view of a Plate Matrix 39. FIG. 63 shows the proper alignment ofthe plate matrix 39 superimposed above the Active Sensing Array 20.

Protrusion 30: a rigid bump of plastic, metal, wood, glass, or othersuch material that is positioned above or below a sensing element 26 onthe Active Sensing Array 20 of that sensing element and whose purpose isto focus force onto the active area 27 of that single sensing element26. The side of the protrusion facing the Active Sensing Array 20 mustbe a shape whose contact with the Active Area of its correspondingsensing element would lie exactly upon or inside of the Active Area ofthat Sensing element 27. Protrusions are arranged in a Protrusion Matrix43 which may be a constituent of an Integrated Plate and ProtrusionLayer (IPPL) 36 or part of an Integrated Protrusion and Base Layer 42.

FIG. 64 Shows the Top View of a Protrusion 30 Properly Aligned Upon theCorresponding Sensing Element 26 on the Active Sensing Array 20.

FIGS. 65A-65F shows the side view of six examples of contact betweenprotrusions 30 and active area of sensing element 27. In FIGS. 65A, 65B,65C, and 65D, examples are shown of protrusions 30 whose contact withthe active area of its corresponding sensing element 27 lies exactlyupon or inside of that active area 27. In FIGS. 65E and 65F, theprotrusions 30 have contact that extend beyond the active area 27 of thecorresponding sensing elements 26 and thus are not appropriateprotrusion configurations for this invention. In the case in FIG. 65D,the protrusion 30 above that sensing element has discontinuous aspectssuch that each of these aspects might be attached to different platesthat meet at that sensing element 26.

Protrusion Matrix 43: A plurality of Protrusions 30 spatially aligned tocorrespond with the sensing elements 26 on an Active Sensing Array 20. AProtrusion Matrix 43 may be a constituent of an Integrated Plate andProtrusion Layer (IPPL) 36 or of an Integrated Protrusion and Base Layer42. FIG. 62A shows the top view and FIG. 62B shows the side view of aProtrusion Matrix 43.

FIGS. 61A, 61B, 62A and 62B are drawn a juxtaposed as a Plate Matrix 39and a Protrusion Matrix 43 respectively would be aligned with eachother.

FIG. 66A shows the Bottom View. FIG. 66B shows the Side View, and FIG.66C, shows the Top View Top of the superposition of a properly alignedPlate Matrix 39 and Protrusion Matrix 43.

FIG. 67 shows a Cut out view of the superposition of a properly alignedPlate Matrix 53 and Protrusion Matrix 43.

Outer and Inner Directions/Side/Face: A sensor may be placed on a table,wall, ceiling or moving object. As a result, referring to top/bottom orup/down is ambiguous. For clarity, use ‘Outer’ to designate theside/direction/face upon which the force is being applied and ‘Inner’ todesignate the opposite side/direction (towards the base of theapparatus). For example in the FIG. 68A showing the device as it wouldbe oriented on a flat surface and 68B showing the device as it would beoriented on a wall, the imposed force 34 is applied to the outer face ofthe Flexible Touch Layer 38. Similarly, the inner face of theprotrusions in the IPPL 36 rest on the outer face of the Active SensingArray 20 such that the inner face of the protrusions 30 are aligned withthe sensing elements 26 on the Active Sensing Array 20. The inner faceof the Active Sensing Array rests upon the outer face of the base layer47. In FIGS. 68A and 68B the Outer Direction 28 and Inner Direction 29are designated with arrows. In any cases of ambiguity, the canonicalorientation is with the sensor placed on a flat surface parallel to thefloor, such as on a table top with the force coming from above, as inFIG. 68A.

Integrated Plate and Protrusion Layer (IPPL) 36: A part containing botha Plate Matrix 53 and a Protrusion Matrix 43, such that the protrusionsare physically connected to adjacent plates on the inner surface. Theprotrusions 30 extend beyond the inner surface and are spatially alignedto correspond with the sensing elements 26 on an Active Sensing Array20. This part may be made of plastic, metal, wood, glass, or other suchmaterial that is rigid or semi-rigid. Methods for fabrication of thisare described below. FIG. 69 shows an embodiment of an Integrated Plateand Protrusion Layer 36.

In various embodiments, the Integrated Plate and Protrusion Layer 36 mayhave some of the shapes depicted in FIGS. 70-73. In all of theseembodiments, there are slits between the plates, but the shapes of theprotrusions 30 vary; the width of the slit may vary as seen comparingFIG. 70 and FIG. 73; the protrusion may continue through the junction tobe flush with the plate as seen comparing FIG. 70 and FIG. 71; or may betapered/trapezoidal towards the inner face of the protrusion as seencomparing FIG. 70 and FIG. 72. FIG. 74 shows a top view corresponding tothe FIG. 70 or FIG. 72 embodiment with slits. FIG. 75 shows a top viewcorresponding to the FIG. 71 embodiment with protrusions continuing tobe flush with the plates. FIG. 76 shows a top view corresponding to FIG.73 embodiment which has a wider slit than the embodiments shown in FIG.70 and FIG. 74. In each of the embodiments shown in FIG. 74-76, the slitalong the edges of the plates, but not at the protrusions, go completelythrough the material.

Corner Protrusion 54: In one embodiment, the protrusion 30 over asensing element 26 on the Active Sensing Array 20 may be contain severaldiscontinuous aspects with each discontinuous aspect attached at thecorner of one of the several plates 35 meeting at that sensing element26 and over that sensing element 26. A Corner Protrusion 54 is definedas one of these discontinuous aspects. With Rectangular plates meetingat a sensing element, up to four Corner Protrusions 54 may impart force,acting collectively as the protrusion 30, upon that sensing element 26as seen in FIGS. 77A-77C, 79, and 80.

FIGS. 77A-77C shows examples of one, two and three corner protrusions54, respectively, lying above the active area 27 of a marked sensingelement 26. In each of these examples, the set of corner protrusions 54together would be considered the ‘protrusion’30 above that sensingelement 26.

In another embodiment of the IPPL 36, such as the one described belowusing compression molding, protrusions 30 may each consist of a set ofcorner protrusions 54. In this embodiment, the outer surface of the IPPLwould be flat, designated as a Flat Top Integrated Plate and ProtrusionLayer 41, allowing, in the case of a single tile sensor, the Flat topIPPL 41 to also function as the Flexible Touch Layer 38.

FIG. 78 shows the Side view of Flat Top IPPL 41 embodiment with plates35 having Corner Protrusions 54 and the outer surface being flat.Protrusions 30 where corners of plates meet will consist of sets ofCorner Protrusions 54 from different plates 35. In this embodiment thesurface is flat with a thin amount of additional material connecting theseparate plates, as seen in the in FIG. 78 and in outer view in FIG. 79and in inner view in FIG. 80. Unlike the embodiments seen in FIGS.70-76, the slits do not continue through between the plates, but insteadform grooves from the inner face as seen in FIGS. 78-80. In such anembodiment, the thickness of such connecting material (between the outerface and the inner edge of the groove) must respect the requirements fora flexible touch layer 38. For example, for a 1 mm thick plate of ABSplastic, and 0.1 mm for the connecting material.

In embodiments of a Flat Top Integrated Plate and Protrusion Layer 41,either a shared coherent protrusion or a set of corner protrusions maybe used (as shown in FIGS. 78-80, corresponding to each sensingelement).

Plate Matrix Layer 53: A part containing a plurality of Rigid Plates 35such that the plates are connected either with a thin flexible top orbottom material or with material in the grooves between the rigidplates. Unlike the IPPL 36, the protrusions 30 are not part of thiscomponent. This part may be made of plastic, metal, wood, glass, orother such material containing methods for fabrication of this aredescribed below. FIG. 81 shows a Flat Top Plate Matrix Layer 116embodiment of a Plate Matrix Layer 53 with thin flexible top materialwhose construction similar to the Flat Top IPPL 41 but without theprotrusions that would be found in the Flat Top IPPL.

Integrated Protrusion and Base Layer 42: A part containing a ProtrusionMatrix 43 and a supporting base 47, such that the protrusions arephysically connected to base 47 on the inner surface, as seen in FIG.82. This part may be made of plastic, metal, wood, glass, or other suchmaterial that is rigid or semi-rigid. Methods for fabrication of thisare described below. In earlier embodiments described in this patent,such as the one shown and described from FIG. 19, are examplescontaining an Integrated Protrusion and Base Layer.

Three cases are shown in which the plate is, respectively: Sufficientlyrigid shown in FIG. 83; sufficiently semi-rigid shown in FIG. 84; andinsufficiently rigid allowing force to be transmitted to the base ratherthan the protrusions shown in FIG. 85. In each case, the externallyimposed force 34 upon the plate 35 is transmitted to different locationson the base layer 47 as the depicted transmitted force 56. FIG. 83 andFIG. 84 represent “Valid Amount of Plate Rigidity relative to theProtrusion Heights”, with the transmitted force 56 being focusedexclusively through the protrusions 30 to the base layer 47. In FIG. 85,the plate 35 does not have a Valid Amount of Plate Rigidity relative tothe Protrusion Heights because it deforms such that some force 56 isimparted on the underlying base surface in a region not through aprotrusion 30. Comparing FIG. 83 and FIG. 21 shows an advantage of theembodiment involving plates 35. Unlike the embodiment shown in FIG. 21with a Semi-Rigid touch layer 31, the plate 35 can be rigid force is nottransmitted to protrusions that it is directly above.

Valid Amount of Plate Rigidity relative to the Protrusion Heights: Aplate has a “Valid Amount of Plate Rigidity relative to the Protrusionheights” if an externally applied force of the outer face a plate wereto result in pressure being applied exclusively to the correspondingprotrusions at its corners, in particular no force is imparted to thesurface between the protrusions; The Plate 35 would not have a ValidAmount of Plate Rigidity if the same externally applied force were tocause the Plate 35 to deform to sufficient extent that the Plate 35would physically come into contact with the region of the Base Layerbetween those four protrusions 30, thereby dissipating force ontoinactive regions of the Active Sensing Array 20. This unacceptable casecan be seen in FIG. 85 where the plate 35 deforms in the middle in anarc the full height of the protrusion 30 allowing the plate to touch thebase. For example, in the case where the protrusions are spaced at 12mm, a 0.5 mm thick rectangular piece of rubber would not have a validamount of plate rigidity to serve a Plate. The distance of thedeformation of the plate materials can be described byE(bend)=L³F/(4wh³d), where L is the length, w and h are the width andheight, F is the applies force and d is the deflection to the load onthe surface.

Flexible Touch Layer 38: This is the outer most layer that is exposed tothe user for direct contact/touch. It is comprised of a thin flexiblesheet of material (e.g., rubber, Teflon, or low density polyethylene.)It must be sufficiently flexible (i.e., having a Young's modulus andthickness such that the stiffness is an order of magnitude less thanthat of the plates—the stiffness of most materials is determined largelyby the product of the materials Young's Modulus [constant to thematerial] and the cube of the material's thickness as in the equationbelow, such that force applied to the surface is primarily transmittedto the plates below the force. In one embodiment, it could be made of0.005″ polyester film.

The stiffness of a material may be computed as per: D=Eh³/(12*(1−v²)),where E=Young's Modulus; h=material thickness; D=stiffness; v=Poisson'sConstant of the material.

The total size and shape of the Flexible Touch Layer 38 can be made soas to match the total size and shape of the networked grid of sensortiles.

Base Layer 47: This inner most layer is a flat featureless sheet lyingbelow the rest of the assembly. In an embodiment where the apparatus 1will lie flat against a flat solid surface, such as a 3″ thick flatglass table, the base layer does not necessarily need to provide rigidsupport as this will be provided by, for example, the table. In anembodiment of an apparatus 1 that would not lay flat on a surface, or ona surface that is not solid, such as a mattress, it would need to berigid, such as a ¼″ thick acrylic sheet.

Adhesive Layer 40: An adhesive layer may be used to affix therespectively abutting functional layers. In one embodiment, the adhesivelayer could be a double sided adhesive film sheet, such as GraphixDouble Tack Mounting Film. In other embodiments a spray adhesive may actas the Adhesive layer used to bond these layers.

Step by Step Description of Internal Working:

FIG. 86 shows a Cross Section of Force Distribution: Flexible TouchLayer 38, Integrated Plate and Protrusion Layer 36, Active Sensing Array20, Base layer 47, Externally Applied Touch Force 34. The Ippl 36Contains Plates 35 and Protrusions 30. The Protrusions 30 are Alignedwith the Sensing Elements 26 on the Active Sensing Array 20.

Internal operation begins when fingers or other objects impose downwardforce 34 upon outer surface of the Flexible Touch Layer 38, as seen inFIG. 86.

This force is then transmitted through the Flexible Touch Layer 38 tothe Plate 35 underneath the force 34 in the Integrated Plate andProtrusion Layer 36.

The respective downward force 34 on each plate 35 of the IPPL 36 isredistributed to the protrusions 30 in the IPPL 36 that are under theplate's 35 respective four corners. The protrusion at any corner of aPlate 35 is shared by up to three other adjacent plates 35. In the casewhere force is concurrently applied to adjacent plates 35, the combinedforce from those adjacent plates 35 are concentrated onto respectiveshared protrusions 30 and measured at the sensing element 26 that thisshared protrusion 30 is in contact with.

Each protrusion 30 at the four corners of a rigid plate 35 is alignedabove a respective sensing element 26 on the Active Sensing array 20,concentrating the force applied to each rigid plate 35 to the activearea of the sensing elements 27 at the plate's corresponding fourcorners.

This creates a concentration of force that is transmitted to the portionof the Active Sensing Array 20 where each protrusion 30 is in contactwith a corresponding sensing element 26, thereby creating a force thatcompresses together the two areas of FSR material 24 in mutual contactat the regions of the Active Sensing Array 20 that comprise the sensingelements 26 (where one FSR 24 region on the outer conducting line of 23the Active Sensing Array 20 is in contact with a corresponding region ofFSR material 24 on the inner conducting line 23 of the Active SensingArray 20 as seen in FIGS. 10 and 11).

As described earlier, this compression creates an increase of electricalconductance between those two areas of FSR material in mutual contact.As the sensor's micro-controller scans through the Active SensingArray's array of sensing elements, each of those changes in conductanceis measured as a change in voltage, which the micro-controller detectsvia an A/D converter that the micro-controller then encodes as a digitalsignal. The micro-controller then sends this digital signal through theUSB to the host computer.

This configuration of components forms a mechanism for even forceredistribution from the Plates to the sensing elements on the ActiveSensing Array whereby a continuous change in position of a touch on theouter face of the Flexible Touch Layer results in a correspondingcontinuous change in the relative force applied to those sensingelements that are nearest to that touch. Those relative forces, whensent to the host computer as part of the data image, permit the hostcomputer to accurately reconstruct the centroid position of the touchthrough arithmetic interpolation.

FIG. 87 shows a schematic view of interpolation: All externally applieddownward force 34 impinging upon the Flexible Touch Layer 38 istransmitted to the plate 35 in the IPPL 36 abutting that force. Theforce 34 on that plate 35 will be focused on the 2×2 nearest protrusions30 on the IPPL 36. Therefore in the Active Sensing Array layer 20 all ofthe force will be concentrated on the 2×2 corresponding active areas 27for the where there is direct mechanical contact with these fourprotrusions 30 and thus mechanically distributed to the respectivesensing elements 26.

The difference between this process using plates 35 and a flexible touchlayer 38 and the similar process that was described without plates butwith a semi-rigid touch layer 31 is that by allowing for a thinner TouchSurface 38 and distinct plates 35 under that touch surface, the localforces on the Flexible Touch Layer 38 are nearly exclusively conveyed tothe plates 35 under that force and then transmitted through thecorresponding protrusions 30 onto the appropriate sensing elements 26.Additionally in this Active Sensing Array 20 is affixed onto a flatsurface and thus cannot deform as might occur in the method withoutplates.

The electronic measurement and processing of the force upon the ActiveSensing Array is identical to that in the method without plates.

FIG. 52 shows an exploded view of the Layers and Assembly in theprototype single tile embodiment using an Integrated Plate andProtrusion Layer (IPPL) with Flexible Touch Layer 38; Integrated Plateand Protrusion Layer 36; Active Sensing Array, 20; Base Layer 47. Whenthe layers are placed into contact, each protrusion 30 in the IPPL 36 isaligned to be in contact its corresponding active sensing area 27 on theoutside surface of the Active Sensing Array 20. An Adhesive Layer 40 wasused between each of the above layers in this prototype embodiment.

Flexible Touch Layer 38: 5 mil Polyester Film

Integrated Plate and Protrusion Layer 36: 31×31 grid of plates with32×32 grid of protrusions. A Custom SLA (Stereolithography) RapidPrototyped part manufactured with Somos 11122 (Clear PC Like) createdwith a supplied CAD file with the IPPL 36 Geometry using standard SLAmanufacturing.

FIG. 88 shows the plate and protrusion dimensions used in the prototypeembodiment of the Integrated Plate and Protrusion Layer 36 in a crosssection view. The plates 35 and protrusions 30 are square, so thesedimensions are the same for both the width and length (not drawn toscale).

Note: In a single tile assembly there are (N−1)×(M−1) plates for an N×Mgrid of protrusions for an N×M Active Sensing Array because there is noneed for a spanning plate between abutting tiles. For example in FIG.52, a 4×4 grid of plates are supported by a 5×5 grid of protrusions andused with an Active Sensing Array with a 5×5 grid of sensing elements.

Active Sensing Array 20: Custom printed sensor, as per description inthe other earlier described embodiments, with a 32×32 grid of sensingelements spaced at ⅜″. Each sensing element has a 4×4 mm overlapping FSRarea. 100 kOhm FSR Ink was used in the ASA.

Base Layer 47: CPVC Sheet, 1/32″ Thick. Note that this embodiment wasone in which it was expected that the apparatus would be placed on asolid table top for use as in the embodiment of the Base layer where theapparatus 1 will lie flat against a flat solid surface.

Adhesive Layer 40: Graphix Double Tack Mounting Film. Three adhesivelayers 40 are used in this assembly.

In this prototype assembly,

a) One side of an adhesive layer 40 is affixed to the inner surface ofthe Flexible Touch Layer 38.

b) The opposite side of that adhesive layer 40 is affixed to the outersurface of the IPPL 36.

c) One side of a second adhesive layer 40 is affixed to the outersurface of the Active sensing array 20.

d) The opposite side of that adhesive layer 40 is affixed to the innersurface of the IPPL 36 such that the protrusions 30 on the IPPL 36 arealigned with the corresponding sensing elements 26 on the Active SensingArray 20.

e) One side of a third adhesive layer 40 is affixed to the inner surfaceof the Active Sensing Array 20.

f) The opposite side of that adhesive layer 40 is affixed to the outersurface of the Base Layer 47.

Pressure Data for this IPPL Prototype Assembly

In the following tests, calibrated weights were placed above a wireintersection. A small rubber cylinder that weighed 5g was used toconcentrate the force at the intersection.

IPPL Sensor

Weight(g) Value From Sensing element(*) 20 30 40 95 60 150 80 200 100260 120 320 140 340 160 380 180 410 200 425 250 480 (*)In the prototypeembodiment here, these are the values measured from the A/D circuitry ofthe PIC24 chip and based on voltages. The values are measured as 12-bitnon-negative values.

Methods to Manufacture the Integrated Plate and Protrusion Layer

In one embodiment, a metal mold can be created for the IPPL usingindustry standard techniques for making molds for plastic parts. TheIPPL parts can be manufactured via injection molding out of ABS plasticusing standard injection mold and molding techniques.

Another way to manufacture a IPPL is to perform selective photo-etchingon both sides of a sandwich that has been formed by affixing thin metalplates, such as 0.005″ thick brass, to both sides of a plastic sheet,such as 0.003″ thick Mylar or kapton, that has been coated with adhesiveon both sides. One of the metal plates will form the plates layer, andthe other will form the protrusions layer. In both cases, the parts ofthe metal plate that should not be etched away are covered with apattern of photo-resist (such as a pattern of toner transferred from alaser printer). Equivalently, the plates can be formed from a standardphoto-polymer such as DuPont Cyrel or BASF Nyloflex, which is firstselectively cured by being exposed to a pattern of UV light, which inthe standard process is in the range of 365 nm after which the unexposedportion is washed away.

Templates for the photo-resistive ink patterns of the two plates can beseen in the FIGS. 89A and 89 B. FIG. 89A shows a photo-resistive inkpattern of the plates' side. FIG. 89B shows a photo-resistive inkpattern of the protrusions. In the embodiment where the plates arephoto-polymer, the negative of these patterns is used.

Another method for creating an integrated plate and protrusion layer 36part is to photoetch the surfaces of two thick flat metal plates, suchas steel plates, so that they form negative relief patterns. A plasticthat is soft when hot yet hard when cool is then placed between thesetwo metal plates, preferably in the presence of a vacuum. The plates areheated and pressure is applied to force them together, thereby creatinga relief pattern in the plastic, whereby the soft plastic is deformedaway from the groove areas to fill the protrusion areas.

The photo-etching is done so as to create smooth slopes in the platerelief pattern, thereby facilitating the subsequent process of pressingthe relief pattern into the plastic.

The arrangement of the two metal plates is shown in cross section in theFIG. 90A below. The top compression plate 57 which creates the groovesin the plastic that define the plate shapes. The bottom compressionplate 58 which creates the protrusions in the plastic. FIG. 90B showsthe resulting groove locations 59, and the resulting protrusionlocations 60.

Another method of manufacture of the IPPL 36 is to create a singlesurface that has a relief structure of both plate shapes as well asprotrusions on only one side, by splitting each protrusion to allow forcontinuous grooves between adjacent squares, as shown in profile view inthe FIG. 91A.

Placing the relief structure which combines the rigid squares and theprotrusions into a part that has a relief structure on only the bottomside confers the advantage that the top of this part will feel smooth toa user's touch. Specifically, this embodiment creates an IntegratedPlate and Protrusion layer 36 that also includes a Flexible Touch Layer38 as in the Flat Top Integrated Plate and Protrusion 41 embodiment partas seen/described in FIGS. 78-80.

One method of manufacturing this relief structure is via compressionmolding of thermosetting plastic. In a variant of this process, theplastic to be compression molded is placed in contact with a thin (e.g.0.003 inch thick) sheet of a flexible plastic such as mylar or kapton.After the compression and curing process of the connected part, thegroove areas will essentially consist only of the flexible plastic 61,with the rigid plastic being located in the plates 35 and protrusions30, as seen in FIG. 91B. This will create the desired mechanicalproperties of rigid plates 35 and rigid protrusions 30, with flexiblehinging between adjoining plates along with an integrated Flexible TouchLayer 38, as in a Flat Top IPPL 41.

FIG. 54 Shows an Exploded View of the Layers and Assembly in thePrototype Single Tile Prototype Embodiment Using a Distinct Plate MatrixLayer 53 and an Integrated Protrusion and Base Layer 42.

with: Flexible Touch Layer 38, Plate Matrix Layer 53, Active SensingArray 20, Integrated Protrusion and Base 42. The grid line intersectionson the Active Sensing Array are the locations of the FSR sensingelements. When the layers are placed into contact, each protrusion 30 inthe Protrusion Layer 42 is aligned to contact the corresponding activesensing area 27 on the inside surface of the Active Sensing Array 20.Additionally, the corners of each plate in the Plate Matrix Layer arealigned to be above the outer active sensing area 27 on the outsidesurface of the active sensing array 20 opposite their correspondingprotrusions. An Adhesive Layer was be used between each of the abovelayers in this prototype.

Flexible Touch Layer 38: 5 mil Polyester Film

Plate Matrix Layer 53: 31×31 grid of plates. 1/32″ Acrylic sheet, customlaser cut to the final shape using two passes. The first pass etchingthe grooves, but not cutting all the way through, at the cornerjunctions. A second pass cutting slits completely through the acrylic toresulting in the part designated in top view FIG. 58A and cross sectionview FIG. 58B. The dimensions used in the prototype are shown in FIG.58A and FIG. 58B (not to scale) which has square plates, so thesedimensions are the same for both the width and length.

Active Sensing Array 20: Custom Sensor as per description in aboveembodiments with a 32×32 grid of sensing elements spaced at ⅜″. Eachsensing element has a 4×4 mm overlapping FSR area. 100 kOhm FSR Ink wasused in the ASA.

Integrated Protrusion and Base Layer 42: 32×32 grid of protrusions, ⅜″spacing, 4 mm diameter hemispherical protrusions. Custom SLA RapidPrototyped part made with Somos 11122 (Clear PC Like).

Adhesive Layer(s) 40: Graphix Double Tack Mounting Film. This hasprotective paper on either side of an adhesive plastic sheet.

In this prototype assembly,

a) One side of an adhesive layer 40 is affixed to the outer surface ofthe Plate Matrix Layer 53, leaving the protective covering on theopposite side intact.

b) One side of second adhesive layer 40 is affixed to the inner surfaceof the Plate Matrix Layer 53, leaving the protective covering on theopposite side intact.

c) Gently bend the Plate Matrix Layer 53 until all the connectingmaterial in the notched grooves at each plate junctions have broken.This leaves a flexible sandwich with the Plate Matrix Layer 53 inbetween two adhesive layers and with each plate no longer rigidlyattached to any other plate.

d) The Active Sensing Array 20 is affixed to the adhesive layer 40(already in place) on inner side of the Plate Matrix Layer 53 using theopposite side of the adhesive layer 40 from step (b). The sensingelements 27 from the Active Sensing Array 20 need to be aligned with theplate junctions on the Plate Matrix Layer 53.

e) One side of third adhesive layer 40 is affixed to the inner surfaceof the Active Sensing Array 20.

f) The Integrated Protrusion and Base Layer 42 is affixed to theadhesive layer 40 (already in place) on inner side of the Active SensingArray 20 using the opposite side of the adhesive layer 40 from step (e).The sensing elements 27 from the Active Sensing Array 20 need to bealigned with the protrusions 30 on the Protrusion Layer 42.

g) The Flexible Touch Layer 38 is affixed to the adhesive layer 40 onouter side of the Plate Matrix Layer 53, using the opposite side of theadhesive layer 40 from step (a).

Pressure Data for this Prototype Assembly

In the following tests, calibrated weights were placed above a wireintersection. A small rubber cylinder that weighed 5g was used toconcentrate the force at the intersection.

Prototype using distinct Plate Matrix Layer 53 and Integrated Protrusionand Base Layer 42

Weight(g) Value from Sensing element(*) 20 0 40 120 60 230 80 320 100420 120 500 140 540 160 570 180 605 200 620 250 650 (*)In the prototypeembodiment here, these are the values measured from the A/D circuitry ofthe PIC24 chip and based on voltages. The values are measured as 12-bitnon-negative values.

Methods to Manufacture the Plate Matrix Layer

One embodiment of the Plate Matrix Layer involves laser cutting asdescribed above.

Other embodiments are analogous to the technique described for theIntegrated Plate and Protrusion layer 36 described above but withoutsteps/facets that create the protrusions.

Methods to Manufacture the Integrated Protrusion and Base Layer 42

In one embodiment, a metal mold can be created for the Protrusion Layerusing industry standard techniques for making molds for plastic parts.The Protrusion Layer parts can be manufactured via injection molding outof ABS plastic using standard injection mold and molding techniques.

Assembly of Sensor with a Thin Base Layer and Co-Planar PCB

FIG. 92 shows an embodiment of a single Stand Alone Tile: Flexible TouchLayer 38; IPPL 36; Base Layer 32; Active Sensing Array 20; PrintedCircuit Board 4.

The embodiment shown in FIG. 92 shows the Active Sensing Array 20 layingflat upon the Base Layer 32, with its Connector Tails 25 connected to aco-planar Printed Circuit Board 4. The base layer 47 in this correspondsto one described earlier where the apparatus 1 will lie flat against aflat solid surface. An advantage of this embodiment is that the entiresensor is thin. For example in the above embodiment, the entire sensoris under 3 mm.

Assembly Involving a Plurality of Tiles

In one embodiment using the Integrated Plate and Protrusion Layer (IPPL)technique, individual tile sensors that are part of grid of sensors arenearly identical to the single tiles described earlier, but may have anextra row and/or extra column of bridging plates 37. In particular, asseen in exploded view in FIG. 93, an individual tile with an N×M ActiveSensing Array 20 and corresponding N×M matrix of protrusions 30 in theIPPL 36 may have an extra row and column of bridge plates 37 in the IPPL36 resulting in an N×M matrix of plates 35. This is unlike the singletile assembly described earlier where the IPPL 36 for such an N×M ActiveSensing Array 20, where there were an (N−1)×(M−1) matrix of Plates 35.Note that there are no protrusions 30 on the additional corners of theseextra bridging plates 37. The Flexible Touch Layer 38 will be a singlecontinuous sheet spanning all tiles in the Grid of tiles.

An example embodiment of an Internal Tile used in a Plurality of tilesthat is based upon an Active Sensing array 20 with a 4×4 matrix ofsensing elements 26 is seen in exploded view FIG. 93, top view FIG. 94,and side view 95. In this example, the IPPL 36 consisting of a 4×4matrix of Protrusions 30. There is a sub-matrix of 3×3 plates 35 withprotrusions at each corner and an additional, an additional row andcolumn of bridging Plates 37 (providing seven additional plates) thathave only some corners resting on protrusions. These bridging plates 37,as described later will span across to share protrusions 30 onneighboring tiles. As seen in FIG. 93 the layers include the FlexibleTouch Layer 38, IPPL 36. Active Sensing Array 20 and Base Layer 47. TheIPPL 36 is aligned such that its 4×4 matrix of protrusions is in contactwith the corresponding 4×4 matrix of sensing elements 26 on the activesensing area. An Adhesive Layer 40 may also be used between each of theabove layers. In this view, the additional row and column of bridgingplates 37 is seen extending beyond the base layer 47, the IPPL 36, andthe active sensing array 20 on two edges.

FIG. 96A and FIG. 96B shows the manner in which adjacent tiles 2 arealigned positioned such that the bridging plate rests on thecorresponding protrusion on its adjacent tile. FIG. 96A shows two tilesbeing aligned. FIG. 96B shows the two tiles properly positioned.

In this embodiment, adjacent Tiles 2 are positioned such that thebridging plates 37 span the protrusions 30 of one tile 2 to theprotrusions 30 of another tile 2. This results in an identicalmechanical distribution of force to the appropriate sensing elements asfor plates spanning protrusions within a tile.

In one implementation of the Base Layer 47, the base can be molded witha cavity on its bottom that could house the sensor tile's PrintedCircuit Board 4, as shown in side view in FIG. 97A and from the bottomin FIG. 97B. Channels would also be molded into the base to supportinter-tile cabling.

In FIG. 97B, this embodiment is seen with the Base Layer 47 has acut-out region 62 on its underside into which the Printed Circuit Board4 securely fits. The Active Sensing Array 20 wraps around two adjacentedges of the Base Layer 32 to electrically connect via the connectortails 23 on the Active Sensing Array 20 to the PCB 4. The IPPL 36 showsthe bridging plate (not to scale). The Flexible Touch Layer 38 spansmultiple tiles. FIG. 97A shows a side view. FIG. 97B shows a perspectiveview as seen from underneath.

In the embodiment with the sensor tile's 2 Printed Circuit Board 4 islocated underneath the device, then the Active Sensing Array 20 must bewrapped around the Base Layer 20 as seen in FIGS. 98A and 98B.

FIGS. 98A and 98B shows the side view of Adjacent Tiles being alignedand positioned. FIG. 98A shows the tile being properly aligned. FIG. 98Bshows the two tiles properly positioned. The Bridging Plate 37 spansprotrusions 30 on different tiles 2. The respective Base Layers 47extend only slightly beyond the last edge protrusion 30. This allows fora gap between the Base Layers 47 that allows the Active Sensing Array 20to wrap around.

In this embodiment, a rectangular grid of N×M tiles, such that thebridging plates span the protrusions of one tile to the protrusions ofanother tile results in an identical mechanical distribution of force tothe appropriate sensing elements as with plates spanning protrusions onthe same tile.

In one embodiment, an apparatus 1 with a grid of tiles 2 can be composedof identical interior tiles 63 and perimeter tiles (north tiles 64. easttiles 65. northeast corner tile 66). FIG. 94 and FIG. 95 show anInterior Tile that has bridging plates on its north and east edge. FIG.99 shows the schematic of tiles being properly aligned. FIG. 100 showsthe tiles in their proper positions with the bridging plates 37 restingupon protrusions 30 on adjacent tiles 2. FIG. 101 shows the tiles intheir proper positions with the Bridge Plates 37 drawn transparently,exposing the Bridging plates 37 on the edge of a tile 2 spanning acrosspairs of protrusions 30 on two different tiles 2 and in the case of thecorner Bridging Plate 37, spanning protrusions 30 on four differenttiles 2.

Interpolation Along Bridge Plates Spanning Tiles

In this embodiment, Bridge Plates 37 span across pairs of protrusions 30on different tiles or in the case of a corner Bridging Plate 37 spanningfour protrusions 30 on four tiles. As there is no mechanical differencein the arrangement of a Bridge Plate 37 on protrusions across multipletiles 2 and for a Plate 35 that spans four protrusions 30 on a singletile 2 regarding to the transmission of force to the respective sensingelements 26, the method of mechanical interpolation is identical forBridge or non-Bridge Plates.

Note that in this arrangement, there is no need for exact registrationbetween the Flexible Touch Layer 38 spanning the plurality of tiles andthe individual sensor tiles, since the Flexible Touch Layer 38 itselfcan be a featureless and uniform sheet of material.

An Embodiment of Apparatus with an N×M Grid of Tiles with SymmetricPerimeter

In one embodiment there may be different types of tiles along the Northand East and Northeast Corner of the grid of tiles as seen in FIG. 102.FIG. 103 shows the North Tiles 64 contain an eastern column of BridgePlates 37; the East Tiles 65 contain a northern row of Bridge Plates 37;the NE Corner Tile 66 does not contain any bridge rows or BridgeColumns; the Interior tiles 63 contain both a northern bridge row and aneastern bridge column of Bridge Plates 35. In this embodiment for a Gridof N rows by M columns of tiles, there would be (N−1)×(M−1) Interiortiles 63, N North Tiles 64, M East Tiles 65 and one NE Tile 66 as seenin FIG. 102.

FIGS. 103-104 show an example with a 3×3 Grid of Tiles with theirrespective Interior 63, North 64, East 65, and NE 66 Corner Tiles intheir appropriate position. FIG. 103 shows a schematic of these tilesbeing properly aligned with bridging plates being aligned with thecorresponding protrusions on the adjacent tiles. FIG. 104 shows thetiles in their proper position.

In other embodiments, all tiles in a grid can be identical. One suchembodiment would have an IPPL 36 with Corner Protrusions 54, as seen inFIGS. 77-80. In this case, the bridging plates would have cornerprotrusions 54 and these corner protrusions 54 would rest upon theactive sensing area 27 of the corresponding sensing element 26 ofadjacent tiles 2.

Interpolation Involving a Plurality of Sensor Tiles

This is the same as described above.

With a networked grid of adjacent sensor tiles 2, the Flexible TouchLayer 38 can consist of a single uninterrupted thin sheet material (suchas 5 mil polyester), which covers all of the sensor tiles 2 in the gridof sensor tiles. This has the advantage that the mechanicalinterpolation process of neighboring sensing elements in the ActiveSensing Layer of different adjoining sensor tiles is identical with themechanical interpolation process of neighboring sensing elements withineach individual sensor tile. The effect from the user's perspective isan interpolating touch response that is exactly equivalent to theinterpolating touch response of a single extremely large sensor tile, asdescribed and seen above FIG. 104. Similarly, the host computer 3, onceit as reconstructed the image from the Tile Topology Table, can treatthe image from a grid of tiles as if it came from a single large sensor.

Note that in this arrangement, there is no need for exact registrationbetween the Flexible Touch Layer and the individual sensor tiles, sincethe Flexible Touch Layer itself can be a featureless and uniform sheetof material.

Non Planar Sensors

In other embodiments, the sensing apparatus 1 may be made to fit upon adevelopable surface, namely one which can be flattened onto a planewithout distortion such as a section of a cylinder as seen in FIG. 105or cone as seen in FIG. 106. Specifically developable surfaces have zeroGaussian curvature.

In one such embodiment, a sensor may be made in the form a section of acylinder as seen in FIGS. 107-111.

FIG. 107 shows an embodiment for an assembly for a ‘Section of Cylinder’Curved Sensor shown from an inside view of the layers. In FIG. 108, itis shown from an outside view. In FIG. 107 and FIG. 108 the layers are:Flexible Touch Layer 38, Active Sensing Array 20, IPPL 3 and Base Layer.

In this embodiment, both the Flexible Touch Layer 38 and Active SensingArray 20 are flexible and can be manufactured similarly to the earlierembodiments. The IPPL 36 may be manufactured via an injection molding asdescribed earlier such that the inner curvature along the plane of theinner faces of the protrusions has the same curvature as the outersurface of the Base Layer 32 which in turn would have its innercurvature matching the outer curvature of the cylinder. Corrections tothis curvature may be made to account for the thickness of the ActiveSensing Array 20, but as the Active Sensing Array 20 is thin and theIPPL 36 somewhat flexible, this correction is not required. FIGS.109-111 show respective views of the IPPL along the height of thecylinder; from the outside; and from the inside, respectively.

In this embodiment, the Base Layer 32 must be sufficiently rigid suchthat the force imparted on the Flexible Touch Layer is not absorbed bydeformation. In one Embodiment, the Base Layer can be made of ABSplastic with the same inner curvature as the outer curvature of a solidmetal cylinder. As seen in the FIG. 112, such a tile 2 would have aninner curvature the same as that of the metal cylinder 67 that it isabutted against.

Non-Rectangular Plates

Sensors may be constructed with non-rectangular plates. For example, inone embodiment, a hexagonal plate matrix 39 as seen in FIG. 113 andcorresponding hexagonal protrusion matrix 43 as seen in FIG. 114 may beused.

A Hexagonal IPPL 36 using the same manufacturing techniques as with therectangular IPPL may be used to create such a part as seen in FIG. 115.

In such an embodiment, an Active Sensing Array 20 with correspondingconductor line 23 spacing so that intersections match the protrusion 30locations of the Hexagonal IPPL 36 may be made, as seen in FIG. 116.

FIG. 117 shows the Hexagonal IPPL seen positioned upon the correspondingActive Sensing Array 20.

In this embodiment, only intersections of grid wires that align withprotrusions from the protrusion Matrix have sensing elements that areused in the mechanical interpolation.

In this embodiment, bi-linear interpolation may be applied to the sixcorners of the plate sensing element values.

Let the six sensors around any hexagonal plate be labeled, in clockwiseorder, A,B,C,D,E,F as in FIG. 118.

One can measure proportional distances between opposite pairs of edgesby the ratios: (A+B)/(A+B+D+E), (B+C)/(B+C+E+F), and (C+D)/(C+D+F+A),thereby defining three lines, each parallel to its associated pair ofedges (one line parallel to AB and to DE, a second line parallel to BCand EF, and a third line parallel to CD and FA). These three linesintersect to form a small triangle in the interior of the hexagon. Thecentroid of this triangle can be taken as a useful approximation to thecenter of pressure applied to the plate.

Fusion

Gesture sensing via a real-time range imaging camera 100 has thefollowing desirable properties: (1) ability to track gestures and (2)ability to maintain consistent identity over time of each finger of eachhand or each part of each foot of each user or each part of each foot ofeach user. Yet range imaging cameras 100 cannot provide high qualitydetected touch 111 and pressure information, while typically operatingat relatively low frame rates.

A pressure imaging apparatus 1 provides low cost, very high frame rate(greater than 100 frames per second), large area pressure imaging. Thedescribed touch-range fusion apparatus technology 104 can, in oneembodiment, combine this pressure imaging apparatus 1 with a newlyavailable generation of low cost real-time range imaging cameras 100 tosimultaneously enable the advantages of both.

Specifically, a range imaging camera 100 tracks every detected touch 111gesture by a user/hand/finger/foot/toe/pen/object, each having a uniquepersistent identifier, while using the pressure imaging apparatus 1 orother touch device 101 to determine positional centroid, pressure (inthe case of pressure imaging apparatus 1) and timing of each detectedtouch 111 with extremely high geometric fidelity and high temporalsampling rate.

Hardware

A Touch-Range Fusion Apparatus 104 can consist of a touch device 101,such as pressure imaging apparatus 1, and one or more range imagingcameras 100 devices. Pressure Imaging Apparatuses 1 are made of modularrectangular pressure tiles 2 that can be seamlessly adjoined to providecontinuous pressure imaging across pressure tiles 2. A Pressure ImagingApparatus 1 can be made in a variety of sizes. Three embodiments includea small device with a 12.5″×17″ form factor, a medium device with a25″×34″ form factor, and a large device with a 50″×68″ form factor.

These three form factors describe the most commonly found finger and peninput non-mobile devices. The small form factor is well suited for asingle user, with sufficient space to use both hands concurrently. Thesmall form factor can be seen in such devices as the Wacom Intuous 4Extra Large and is comparable in size to an average desktop display [8].The medium form factor can be more easily used by multiple participantsand is the size of many interactive tabletop surfaces. For example, theMicrosoft Surface and Diamond Touch are approximately the same size anddimensions as the medium form factor example [9,10]. The large formfactor is primarily seen in collaborative interactions between manyusers at whiteboards as well as for floor sensors that can track thetime-varying pressure across a surface induced by users' feet movements.SMART Electronics produces interactive whiteboards with comparable sizes[11].

One embodiment of a range imaging camera 100 contains a IR Range Camera106 and, optionally, an RGB camera 103. Tracking of object features isdone primarily from range data. The RGB camera 103 can be used forassisting in identifying objects in the 3D space, while also providinguseful visual feedback to users of the device.

FIG. 122, FIG. 132 and FIG. 133 show three different possible placementsfor range imaging cameras 100 for desks/tables/walls, and FIG. 128 showsa possible placement appropriate for floors.

In one implementation, one or more range imaging cameras 100 are placedin key areas around an pressure imaging apparatus 1 to achieve the mostefficient and cost-effective means of accurately identifying fingers,feet, pens and objects in 3D space. The location and number of camerasare chosen so as to limit occlusion issues, and to maximize pixel/depthresolution as needed for accurately identifying features.

Identifying Fingertips, Palms, Parts of Feet, Pens and Objects in 3DSpace

Using the range imaging camera 100 data, fingertips, palms of hands,parts of feet, pens and objects are identified using image analysisprocess algorithms such as [1], [2], [3]. [4]. [5]. [15], [16], [22],[23] or using any other image analysis process which is standard in theart. To begin, feature extraction is performed on the data from therange imaging cameras 100. This can be done, possibly in conjunctionwith supplementary information from the RGB cameras 103, in order toextract information about shape, including lines, edges, ridges,corners, blobs and points. 3D shape recognition provides high confidenceinformation to the feature recognition. This information is passed tomachine learning algorithms, trained on various stimuli to identify handskeleton features, finger tips, foot shape, pens and objects. Once theobject has been identified, the location in 3D space of the objectfeatures is tagged. The identity and xyz position of each feature isused to determine whether a given object or feature is in contact withor off the pad when tracking blobs on the pressure imaging apparatus 1or other touch devices 101.

Because the Touch-Range Fusion Apparatus 104 can have more than onerange imaging camera 100, this analysis software composites theidentified features from all angles in order to give a complete list ofobjects within the scene to the software that will perform the fusioncalculation that maps identified 3D objects to detected touches 111 uponthe surface.

An added benefit to identifying finger tips, pens and objects is thatpalms, wrists and unwanted items can be rejected when tracking objectson the touch device 101. If an application, for instance, requires onlypen input, then all other identified objects can be rejected.

Mapping Fingertips, Feet, Pens and Objects to Tactonic Device 101Contacts

In one case, when an object touches a Pressure Imaging Apparatus 1, ananti-aliased image of pressure is given. This pressure image is used tofind the centroid 107 of a fingertip, pen or object. Each centroid 107can be tracked continuously across the entire Pressure Imaging Apparatus1. This accurate centroid 107 data is used, along with the identity ofobjects derived from Range Imaging Camera 100 data, described above, togive each centroid 107 an identity that can persists even when thatfinger or object loses contact with the surface. Alternatively to apressure imaging apparatus, a touch device 101 can be used that tracksthe centroid 107 of each detected touch 111 upon the surface, althoughpossibly without tracking pressure.

The identity of each centroid 107 is obtained by searching through thelist of identified objects and features identified in the by the Rangeimaging camera 100 data, as described above. If the object/feature islocated near the touch device 101 plane and above the location of thecentroid 107 in the X-Y position, then the centroid's 107 identity canbe obtained.

Contacts made to the touch device 101 are identified and trackedcontinuously as objects and hands and feet move around the device. Thiscontact data can be used for more robust tracking of persistentidentity. In particular, if the identified contact becomes obscured fromthe range imaging cameras 100 because of occlusion, then the contactwill retain its identity as long as the object remains in contact withthe touch device 101. If initial contact is made in a region that isobscured from the range imaging camera 100, then contact identity can bemade when the object/feature reveals itself to the range imaging camera100.

Support for Simultaneous Multi-User Collaboration

Distinguishing between individual users 109 becomes important in largerform factors when multiple participants are using a space concurrently.Each individual user 109 is identified by looking at the entranceposition of the arm, the angle of the arm, and continuous tracking ofindividual users 109 as their arms and hands move around the visiblearea. Similarly each individual user 109 is identified by continuallytracking the position and orientation of each participant's body, legsand feet as they walk around upon a touch device 101 floor surface. Aseach foot or hand and stylus moves across the touch device 101, itsindividual user 109 identification is maintained.

For example, FIG. 123 shows a Left Hand 118 and Candidate Right Hand-A119 which is within the individual user maximum reach 108, so the twohands may belong to the same individual user 109. Candidate Right Hand-B120 is beyond the individual user maximum reach 108 of Left Hand 11, soLeft Hand 118 and Candidate Right Hand-B 119 must belong to differentindividual users 109.

Applications Enabled by the Invention

In addition to new unique gestures available by fusing range imaging 100and touch device 101, existing gestures for range imaging cameras 100and touch device 101 are also supported. Application support softwaremaps gestures performed on the device to actions and keystrokes on thecomputer. Along with the control panel, applications and plug-ins thatthis technology supports includes musical instrument emulation,simulated surgery, simulated painting/sculpting, athletic games andactivities that depend not just upon body movement but also on shifts inweight and balance, and other applications that require a combination ofisotonic and isometric control can be implemented to attain the fullcapability.

Uses for the Invention:

Interactive Whiteboards: According to Futuresource Consulting, Ltd.market report for this sector, 900K Interactive Whiteboards were sold in2010, up from 750K in 2009, mostly in the Education sector. A typicalInteractive Whiteboard consists of a short throw projector displayingonto a large touch device 101 (for example to 6′×4′). The current modelsfor these large format touch device 101 utilize a set of optical camerasalong the perimeter to track user detected touches 111 and gestures.While that approach can provide limited multi-touch and multi-usersupport, it cannot identify the user, hand or finger of detected touches111. Additionally, actions may be occluded by the presence of multiplehands in the camera path. Beyond the significantly greater gesturevocabulary achievable from robust hand action tracking and the addeddimension of pressure, the sensor fusion approach also addresses theeducational need for robust at-board multi-student interaction andcollaboration.

Personal Desktop Peripheral: A personal desktop peripheral represents ageneric Computer Human Interface (CHI) technology which, like the mouseor keyboard, is application blind. While many types of applicationscould be created to take advantage of robust gesture vocabulary, apregnant initial application market for this desktop peripheral would bea game controller. Computer games focus on providing vivid graphicalexperiences with compelling game play. Computer gamers are bothcomfortable and fluent with user input devices that manipulate iconicrepresentation of their character and controls while looking at thevideo display (and not at the input device). The Microsoft Kinect,introduced in November 2010, sold 10M units in its first 60 days, yet itdoes not provide the level of controlled precision or responsivenessrequired for many games, such as first person shooter games. Kinectprovides relatively coarse positional accuracy and low camera framerate. For instance, the Kinect has a frame rate (30 fps) that is aquarter as responsive as keystroke input scanning (125 Hz). Thetouch-range fusion apparatus 104 would provide a broad canvas for gamecontrol with extremely accurate control and response for surfaceinteraction as users touch and press upon surfaces with their hands andfeet.

List of Components

Range Imaging Camera (RIC) 100: produces a 2D image showing the distanceto points in a scene from a specific point, which is implicitly a pointin 3D. There are many types of Range Imaging Cameras 100 commerciallyavailable using well established techniques such as: Stereotriangulation, Sheet of light triangulation, Structured light,Time-of-flight, Interferometry, and Coded Aperture. In one embodiment aMicrosoft Kinect peripheral can be used as the Range Imaging Camera 100.The Kinect contains a PrimeSense Range Imaging Camera 100. There areopen source APIs available to utilize this camera in the Kinect, such asOpenCV, as well as the Microsoft Kinect API. While the Kinect also hasan RGB Camera 103 which can be used in conjunction with this invention,the RGB camera 103 is not used as a required component in thisinvention. In the Kinect Embodiment, there is a standard USB cable 9.FIG. 124 shows a range imaging camera 100 with an IR camera 106, a RGBCamera 103 and a USB cable 9.

Touch Device (TD) 101: A touch device 101 that is able to detect andtrack detected touches 111 on a surface. There are many well establishedtechniques for touch devices 101 as well as a multitude of commercialdevices, such as the Apple Magic Mouse. The Magic Mouse embodimentincludes a standard USB cable 9. Similarly there are ubiquitous smartphones and tablets, such as the Apple iPhone or iPad that contain TouchDevices 101. Embodiments of touch devices 101 include those using:Resistive, Projective Capacitive, Optical, and Frustrated Total InternalReflection (FTIR) methods of operation.

FIG. 125 shows a Touch Device 101, Such as the Apple Magic Mouse, with a(1) Touch Device 101 and (2) USB Cable 9.

Pressure Imaging Apparatus 1: is a Touch Device 100 that also providespressure data at surface contact along with positional detected touch111 data. An embodiment of a Pressure Imaging Apparatus 1 includes astandard USB cable 9. Other embodiments of Touch Devices 101 thatprovide some degree of pressure data (although with less accuracy ofpressure sensing than a pressure imaging apparatus 1) include FTIR.

FIG. 126 shows Pressure Imaging Apparatus 1 with a USB Cable 9.

Computer 3: A computer 3 or other device with a microprocessor with ameans for receiving data from one or more touch device 101 and one ormore Range Imaging Camera 100. An embodiment of a computer 3 is aMicrosoft Windows based Computer.

Step by Step Description of User Experience:

FIG. 127 shows a Table Top Embodiment with a Touch Device 101, RangeImaging Camera 100 Physical Objects 102 such as User's Left 118 andRight 121 Hand.

FIG. 128 shows a Floor Embodiment with a Touch Device 101, a RangeImaging Camera 100, Physical Objects 102 such as Individual Users 109.

From the user's perspective, operation is as follows:

In one time step, one or more users' hands or other physical objects 102are within the field of view of the Range Imaging Camera 100. Acontinuous image from the Range Imaging camera 100 is transmitted to thecomputer 3. Concurrently any user may impose a finger, hand palm, toe,foot, knee, other body part, or other physical object onto the top ofthe touch device 101. A continuous image of this imposed touch istransmitted by a touch device 101 to a host computer 3.

On the computer 3 the Range Image of spatially varying depth is storedin a region of computer memory. From there computer software on thecomputer 3 can be used to store the image in secondary storage such as adisk file, to display the image as a visual image on a computer display,to perform analysis such as construction of a hand object model 105,hand tracking, body model, body tracking, foot shape model, foottracking, region finding, shape analysis or any other image analysisprocess which is standard in the art [1-5], or for any other purpose forwhich an image can be used.

In an embodiment with a pressure imaging apparatus 1, on the computer 3the image of spatially varying pressure is stored in a region ofcomputer memory. From there computer software on the host computer canbe used to store the image in secondary storage such as a disk file, todisplay the image as a visual image on a computer display, to performanalysis such as hand shape recognition, finger tracking, footstep shaperecognition, footstep tracking, region finding, shape analysis or anyother image analysis process which is standard in the art, or for anyother purpose for which an image can be used.

On the next time step, the above process is repeated, and so on for eachsuccessive time step.

Outside Operational Point of View

FIG. 129 shows an embodiment of a Touch Device 101, Range Imaging Camera100, USB Cable 9 from Touch Device 101 to a Computer 3, a USB Cable 9from a Range Imaging Camera 100 to Computer 9, and Computer 3.

One or more Touch Devices 101 and one or more Range Imaging Cameras 100are connected to a Computer 3.

Each Touch Device 101 has one or more of the Range Imaging Cameras 100aimed at its surface.

Each Range Imaging Camera 100 is calibrated/registered with the TouchDevice(s) 101 that it is aimed at. This is done using well establishedsoftware techniques such as algorithms described in [17], [18], [19],[20], [21], or any other image analysis process which is standard in theart. A direct result of this calibration/registration in a well definedmapping of points on the 2D Touch Devices 101 to points in the 3Dcoordinate system of the Range Imaging Camera 100.

Internal Operational Point of View

Using image analysis processes on the Range Imaging Camera 100 data,such as [1], [2], [3], [4]. [5]. [15], [16], [22], [23] or using anyother image analysis process, which is standard in the art, objects inthe scene may be identified, mapped to know model types, and tracked in3D space.

Continuous time varying 3D Articulated Models of each hand, full body,or other object with a known geometry, such as a pen, are constructedfrom the Range Imaging Camera 100 data using image analysis process suchas [1], [2], [3], [4], [5], [15], [16], [22], [23] or using any otherimage analysis process which is standard in the art.

Continuous time varying detected touch 111 tracking of finger, palms, orother objects in contact with the Touch Device 101 are constructed fromthe surface data from the Touch Device 101 using detected touch 111tracking process such as [7] or [22] or using any other touch trackingprocess which is standard in the art.

Step by Step Detailed Algorithm how to Combine the 2d and 3d InfoTogether

A plurality of identifiable object models; such as hand, body, pen,ball, cylinder, hammer, or any other object appropriate for anapplication utilizing this invention; are stored as available data ofknown types. This data includes any data necessary for identifying theobject type as well as a geometric skeletal model including a set ofArticulation Joints 112 for this object type and a set of TrackableContact points 110 for that model. For example in a hand object model105, articulation joints 112 would include the wrist and individualfinger joints 112 while the contact points would include the fingertips. For purposes here, the model types are identified as T_(i). Forexample, T₁ may designate the model type for hand, T₂ may designate themodel type for pen, etc. This object identification, mapping andtracking in 3D space can be accomplished utilizing an image analysisprocess such as [1], [2], [3], [4], [5], [15], [16], [22], [23] or usingany other image analysis process which is standard in the art.

FIG. 130 a Shows the Resulting Hand Edge 122 of a Hand Detected UsingRange Imaging Camera 100 Data from a User's Hand 115 Image and afterApplying Standard Art Edge Detection Algorithms; FIG. 130B Shows theHand Edge 122 Overlayed with the Resulting Feature Skeleton of the HandObject Model 105 Derived by Applying Standard Art Algorithms; and FIG.130C Showing the Skeleton of the Derived Articulated Hand Object Model105 Showing the Trackable Contact Points 110 in the Model, Such asFinger Tips and Articulation Joints 112 in the Model, Such as Wrist,Finger Joints, Etc.

As each object is first detected and identified as a known Model TypeT_(i), it will be assigned a unique element identifier, E_(j) which isadded to a list of known Elements in the Scene. Thereupon the systemwill continuously time track the 3D coordinates of each joint in J_(jn)(n indicating the n^(th) Joint 112 number in T_(i)), as well as thecontact points 110, C_(jm) (m indicating the m^(th) contact point 110number in T_(i)), of the element E_(j). Tracking of the Joints 112 andContact Points 110 corresponding to the element's model is maintainedeven when some of the joints 112 or contact points 110 become occluded(either occluded by itself as when fingers become occluded in a clenchedfist, or by another objects in the scene). A contact point 110 will beconsidered occluded if that contact point is not visible by the RangeImaging Camera 100 at a specific moment in time.

FIG. 130D shows an example of articulated model for a hand ElementE_(j), with labeled joints J_(jn), and contact points, C_(jm)

Specifically the computing system will maintain a list of Elements,E_(j); in the scene with the following data:

-   -   Model Type, T_(i)    -   At any point in time:        -   A set of 3D positions, one for each joint 112 J_(jn), in            Global Coordinates ⁽*⁾        -   A set of 3D positions, one for each contact point 110 C_(jm)        -   A set of occlusion Boolean values, one for each contact            point 110 C_(jm), indicating whether that contact point 110            is currently visible from the range imaging camera 100

(*)—As described in a separate section all Positions can be mapped to aglobal coordinate system for the scene.

Concurrently, for each touch device 101 there will be a continuous timevarying set of detected touches 111 on the touch device 101 of objectsin contact with the Touch Device 101 are tracked using a touch trackingprocess such as [7] or [22] or using any touch tracking process which isstandard in the art.

As each detected touch 111 is first detected and identified it will beassigned a unique touch identifier, P_(j) which is added to a list ofknown Touches for that device. As is standard practice in the art [22],if a touch, P_(j), leaves the surface and a new touch is detected withina designated time threshold and distance threshold, that touch will begiven the same id, P_(j) as in the case of a finger tap on standarddevices such as the Apple iPad. A touch that has left the surface anddoes not reappear on the surface within that threshold of time anddistance is considered ‘no longer active’.

FIG. 134 shows a Touch Device 101 with a Set of Contact Points P_(k).

Specifically the computer 3 will maintain a list of Touches, P_(k) foreach device with the following data:

-   -   At any point in time:        -   The 2D position of the touch mapped to 3D Global Coordinates            ⁽*⁾        -   In the case of a Pressure Imaging Apparatus 1, the pressure            value of the touch.

(*) As described in a separate section all Detected Touch 111 Positionscan be mapped to a global coordinate system for the scene.

For each contact point 110 object model type as T_(i) a contact radiusmay be specified as data. For example, the contact radius correspondingto a finger tip would be approximately ¼″, corresponding to the distancefrom the position in the model of the finger tip (inside the finger) tothe surface of the object itself (corresponding to a point on the pad ofthe finger). This contact radius may be scaled to the side of the actualElement as appropriate for the application of the invention. For examplea child's hand is much smaller than a large adult man's hand so thecontact radius for the child's finger might be approximately ⅛″. In oneembodiment, a scaling factor might be computed relative to the distanceof two designated adjoining joints 112.

A Detected Touch 111 P_(k) is no longer active when the detected touch111 has left the surface and has not come in contact again within timeand distance thresholds described earlier (such as a tap motion). In oneembodiment, the time and distance thresholds may match the associatedcontact point 110. For example a foot tap have a larger time thresholdthan a finger tap.

Below is the algorithm for associating Detected Touches 111 with Contactpoints 110 and for associating Contact Points 110 with Touches:

For each time step t

-   -   Obtain the new state of the object elements {E_(j)} in the scene        at time t, derived from the Range Imaging Cameras 100 data.        -   For each new element E_(j) first introduced to the scene in            this time step            -   For each Contact Points 110 C_(jm), of that element                -   Set the Detected Touch 111 associated with that                    Contact Point 110 to ‘none’    -   Obtain the new state of the Detected Touches 111 {P_(k)} at time        t from the Touch System        -   For each new Detected Touch 111 P_(k) first introduced in            this time step            -   Set the Contact Point 110 associated with that Detected                Touch 111 to ‘none’        -   For each Detected Touch 111 P_(k) that has become no longer            active in this time step            -   If there is a Contact Point 110 C_(jm) associated with                this Detected Touch 111                -   Set the Detected Touch 111 associated with C_(jm) to                    ‘none’            -   Remove this Detected Touch 111 P_(k) from the set of                Detected Touches 111    -   For each Detected Touch 111 P_(k) that does not have a contact        point 110 C_(jm) associated with it        -   For each Contact Point 110 C_(jm) that does not have a            detected touch 111 associated AND the Contact Point 110 is            not currently occluded            -   compute the Euclidian Distance d between respective                positions in Global Coordinates of the Contact Point 110                C_(jm) and the Detected Touch 111 P_(k)            -   if d is less than the contact radius for that Contact                Point 110, C_(jm),                -   Associate C_(jm) with P_(k)                -   Associate P_(k) with C_(jm)    -   Display the data, {E_(j)},{C_(jm)}, {J_(jn)}, and {P_(k)} along        with the computed associations (*)    -   Provide this data, {E_(j)}, {C_(jm)}, {J_(jn),}, and {P_(j)}        along with the computed associations (*) via an API to all        higher level systems for further analysis (*)(**)

On the next time step, the above process is repeated, and so on for eachsuccessive time step.

Note:

(*) For Contact Points 110 C_(jm), that have associated Detect TouchesP_(k), positional information from the Detected Touch P_(k) will alwaysbe more accurate than the positional information from the Contact PointC_(jm) (from the Range Imaging Camera 100 data analysis). Specifically,while the position of an occluded Contact Point 110 is either inaccurateor unavailable, an accurate position for any occluded Contact Point 110C_(jm) is available via the position of the associated detected Touch111 P_(k).

(**) The data {E_(j)}, {C_(jm)}, {J_(in)}, and {P_(j) } along with thecomputed associations may be provided to higher level systems forfurther analysis such as gesture synthesis or gesture analysis thatwould extract higher level gestures such as in [27] which in turn couldbe made available for use in an application FIG. 135 shows a BlockDiagram showing the Range Imaging Camera 101 and Touch Device 102connected to the Computer 3. Using the above algorithm, the element data{Ej} is stored in the Computer Memory for Element Data 123 and theDetected Touch Data is stored in the Computer Memory for Detected TouchData 124.

Combine Multiple Range Imaging Cameras and Touch Devices

The ability to combine, over a large multiple user 109 surface, highquality semantic data about hand gesture and hand/finger identificationas well as foot gesture and foot/toe identification with numericallyhigh quality information about the position, exact time and, in the caseof a Pressure Imaging Apparatus 1 pressure of each detected touch 111upon a surface, and to make this data available in an API, will enablenew kinds of interactive human/computer interface applications that wereheretofore unattainable.

The broader impact/commercial potential of this invention follows fromthe combination, over a large multiple user 109 surface, of high qualitysemantic data about hand gestures, foot gestures and object manipulationwith high resolution fine detail from surface data, enabling new kindsof interactive human/computer interface applications heretoforeunattainable, in scenarios where collaborators gather and/or walk aroundin the presence of tables and projection walls to do high qualitycollaborative work using natural and expressive hand, foot andobject-manipulation gestures. This Touch-Range Fusion Apparatus 104approach is superior to approaches using range imaging cameras 100 ortouch device 101 alone, because it allows both isometric and isotonicgestures along with both full hand/finger segmentation and high qualitytouch/pressure sensing. As both range imaging cameras 100 and touchdevices 101 become low priced commodities, costs become sufficiently lowthat this type of touch-range fusion apparatus 104 can be broadlydeployed in homes, offices, schools or other places, to enable people togather and walk around in the presence of tables and projection walls todo high quality collaborative work. This will have strong implicationsfor education, teleconferencing, computer-supported collaborative workand educational games, as well as interactive simulation for scientificvisualization, defense, security and emergency preparedness.

Separately, a novel computer human interaction technology, here called aTouch-Range Fusion Apparatus 104, is described that enables robustgestures and high quality/precise hand/finger input as well as foot/toeinput along with disambiguation of multiple individual users 109, userhands 115, individual fingers, individual feet and toes, pens andobjects over a surface area. Data from range imaging cameras 100 is usedto track movements of hands and feet and to maintain consistenthand/finger and foot/toe identity over time, and this information iscombined with a surface touch device 101 to determine accuratepositional surface information with a high frame rate. This results in aTouch-Range Fusion Apparatus 104 enablement along with a softwareabstraction that reliably combines data from one or more Range ImagingCameras 100 with data from a Pressure Sensing Apparatus 1 or any othertype of touch device 101 capable of detecting the location of one or aplurality of detected touches 111 upon a surface, to create a highquality representation of hand and finger action as well as foot and toeaction for one or more users or for any other object above and upon alarge area surface. This technology enables an inexpensive commercialdevice using only commodity range imaging cameras 100 and touch devices101, and where the pressure imaging apparatus 1 or other type of touchdevice 101 can occur at a data rate that is substantially faster thanthe frame rate of commodity range imaging cameras 100, such as onehundred to two hundred frames per second, along with a softwareabstraction that enables robust hand and foot action/gesture andindividual hand/finger/foot/toe/object identification anddisambiguation.

When used in combination, range imaging cameras 100 and high-frame-ratepressure imaging touch devices 101 suffer none of the deficiencies ofeach technology alone. In particular, combined data from range imagingcamera(s) 100 and a touch device 101 allows a software layer todetermine whether fingertips or pens are touching the surface, tocontinuously track identified fingertips and pens that are touching thepressure imaging apparatus 1 or touch device 101, and to maintain theidentity of touching fingertips and pens even when the target becomesobscured from the camera. In addition, collaboration between multiplesimultaneous users can be supported, in the described invention allows asoftware layer to differentiate multiple individuals that aresimultaneously using the same workspace, and to maintain owner ID onuser hands 115/styli as users' hands cross over each other or, in thepresence of multiple pairs of feet, upon a floor or other surfaceunderfoot.

Using standard art 3D Transformation Matrix techniques, a common globalcoordinate system can be established for multiple Range Imaging Cameras100 and Touch Devices 101. When one or more touch devices 101 are used,a calibration process must be completed in order to obtain thetransformation matrix between the range imaging camera 100 and thesurface of the touch device 101. In one implementation, calibrationcubes 113 are placed at the four corners of one touch device 101. Usingthese corner coordinates, a transformation matrix is determined betweenthe points and the range imaging camera 100. Together, these four pointscreate a surface plane for the touch device 101. This process must becompleted for each touch device 101 in the camera's view. If multiplerange imaging cameras 100 are used, then a transformation matrix isdetermined for each touch device 101 and range imaging camera 100 pair,which proscribes the coordinate transformation between that touch device101 and that range imaging camera 100. In one implementation, thisprocess is repeated for each touch device 101 that is being monitored.If multiple range imaging cameras 100 are associated with a touch device101, then a global transformation matrix can be determined between therange imaging cameras 100, using the touch device 101 as a commonreference coordinate system. Having multiple range imaging cameras 100having overlapping views allows for the position of each subsequentrange imaging camera 100 to be determined during calibration. If aglobal matrix is desired for a range imaging camera 100 that views notouch device 101 with another range imaging camera 100, then that matrixmust be associated with the range imaging camera 100 or the touch device101.

FIG. 131 Shows Cubes Placed at the Four Corners of a Touch Device.

Gestures enabled by fusing touch devices 101 and range imaging cameras100.

Gestures enabled by touch devices 101 and range imaging cameras 100 relyon the identification capabilities of the range imaging cameras 100being paired with the accuracy of the touch devices 101.

Single Touch:

Any gesture made possible by the touch device 101 with a single touchcan be expanded to have a specific action state based on the detectedtouch 111. For instance, if fingers of the hand are being used, theneach finger can have a separate action state attached. This means thatif one hand is used, five separate actions can be performed, one foreach finger, without needing to rely on a menu to switch between theactions. Additionally, single touch objects, such as pens can bedistinguished from fingers to provide alternate interactions or toprevent accidental input.

In one implementation, input from the touch-range fused apparatus 104can be used to emulate a mouse by mapping mouse movement to the movementof the index finger on the touch device 101, left click to the thumbtaps and right click to middle finger taps. This example illustrates theutility of the sensor fusion technique. Without the range imaging camera100, finger-touch identification would be lost and without the touchimaging, accuracy and high frame rate would be lost.

Multi-Touch:

When the scope of interaction is expanded to multiple detected touches111, precision chording is possible. Using a touch device 101 without arange imaging camera 100 limits the possible action states to the numberof inputs. For instance, if fingers of a single hand are used on a touchdevice 101, then only five action states are available (one to fivetouches). When fused with a range imaging camera 100 to identifytouches, chording is possible. Chording is the process of using specificdetected touches 111 simultaneously to perform a gesture. For example,using the thumb and index finger simultaneously could perform a separategesture than the thumb and middle finger simultaneously. Identifyingdetected touches 111 means that (2̂n)−1 action state combinations arepossible for n number of detected touches 111. For instance, thecombination of possible action states for fingers of a single hand goesfrom 5 to 31 when a range imaging cameras 100 are added.

In one implementation, the right hand holds a pen that provides positioninput to a painting program by touching the touch device 101. As theuser draws, the left hand can use specific chording combinations toswitch between 31 set actions states for the pen.

Palms/Hands/Feet/Objects:

Fusing a range imaging camera 100 and a touch device 101 can also beused to reject unwanted input and add action states to non-standardtouch inputs like hands, feet and objects.

When using a touch device 101 by itself, unintended input can occur. Forinstance, a palm can be placed on a touch device 101 and can be confusedfor a detected touch 111. When fused with a range imaging camera 100,the skeleton of the hand is determined which allows the touch to beidentified as a palm and the input can be rejected. The same idea can beapplied to other objects that should be rejected from providing input.For instance, a coffee cup placed on the touch device 101 could berejected.

Hands, feet and objects can also provide alternate forms of interactionthat rely on what a touch device 101 would consider multiple touches.For example, touching with different parts of the palm could be mappedto different action states. Without the range imaging camera 100, theregion of the palm that was touching could not be determined.

Multiple Individual Users 109:

By itself, a touch device 101 cannot distinguish individual users 109that are touching the same device. When paired with range imagingcameras 100, then the individual users 109 can be determined and touchescan be assigned to the correct individual user 109. This allows forsimultaneous interaction from multiple individual users 109 or forcollaborative interactions.

For example, the touch-range fusion apparatus 104 can disambiguatebetween the scenarios of a plurality of simultaneous detected touches111 from different fingers of one hand, a plurality of simultaneousdetected touches 111 from fingers belonging to different hands of thesame user, a plurality of simultaneous detected touches 111 from fingersbelonging to the hands of two different individual users 109.

Similarly, the touch-range fusion apparatus 104 can be used todistinguish between the scenarios of simultaneous detected touch 111upon a sensing floor by two feet of one user, and simultaneous detectedtouch 111 upon the sensing floor by the feet of two different individualusers 109.

Alternate Embodiments of Camera and Touch Device Configurations

In one Embodiment that would be appropriate for tabletop hand gesturetracking would consist of a Range Imaging camera 100 aimed at a narrowangle, such as a 3° onto a 12″×18″ with the camera recessed 6″ away fromthe touch device 101.

FIG. 132 Shows an Embodiment of the Invention with Touch Device 101 andRange Imaging Camera 100.

In another Embodiment the Range Imaging camera 100 can be placed on asupporting stand and aimed down at the Touch Device 101 at a modestangle, such as a 30°. This configuration could be appropriate fortabletop hand gesture tracking onto a 12″×18″ touch device. It couldalso be appropriate for a game controller with a 5′×6′ touch PressureImaging Apparatus 1.

FIG. 122 Shows an Embodiment of the Invention with Touch Device 101,Range Imaging Camera 100, Supporting Stand 114 Allowing the RangeImaging Camera 100 to Face the Touch Device 100 at a Sharper Angle.

In another embodiment that would be appropriate for hand gesturetracking would consist of two Range Imaging cameras 100 can be aimed ata narrow angle, such as a 2° onto a 16×25″ Touch Device 101, as seen inFIG. 133.

Utilities

The following are some utilities for the touch-range fusion apparatus104.

Electronic whiteboard:

Our sensor fusion can be a component of an electronic whiteboard, whichconsists of a flat touch device 101, one or more range imaging cameras100, a computer 3 and a display projector that projects the computervideo images on the surface. The touch-range fusion apparatus 104 servesas the input for the electronic whiteboard. Input can come from a pen orfinger, which is identified by the range imaging camera 100, and draws aline on the electronic touch device 101. The computer uses contact pointdata from the touch device 101 and maps them to pixels on the projecteddisplay image, such as the pixels where the pen's path is being traced.Individual fingers of the user can be placed onto the surface to changethe color of the pen with a separate hand gesture.

Collaborative Surface:

A collaborative surface that uses the touch-range fusion apparatus 104consists of a touch device 101, one or more range imaging cameras 100, acomputer 3 and a projector. In one implementation, multiple individualusers 109 gather around the touch device 101 and touch images that aredisplayed on the surface. Using the location, arm distances and relativearm angles, individuals can be distinguished from each other. When auser makes contact with the touch imaging surface, photos can beselected if the touch lies within the displayed photo. Dragging a fingeralong the surface moves the photo. The location of the user that isholding the photo, which is calculated when determine the distinctiveusers 109, is used to rotate the selected image so that the image isplaced right-side-up for the user.

Computer Peripheral:

A computer peripheral would consist of a touch-range fusion apparatus104 and some communication protocol that passes information to and froma computer 3. It is possible with this peripheral to emulate a mouse.Using the identification of finger tips, the thumb can be mapped tomouse movement, the index finger can be used as a left mouse click andthe middle finger can be used to right click.

Game Controller:

A game controller that uses a touch-range fusion apparatus 104 is madeup of the touch-range fusion apparatus 104 and a communication protocolto a gaming console. Interaction can come from hands, feet, bodies, orobjects. In one instance, multiple individual users 109 dance on a 6foot by 6 foot touch device 101 as a display from the gaming consoleshows dance moves to complete. Each user's foot can be determined byusing the range imaging camera 100 data. Correct steps are rewarded byan increase in score on the game.

References, all of which are incorporated by reference herein:

-   [1] V. Savoy. Kinect Finally Fufills Its Minority Report Destiny.    Engadget. 2010.-   [2] A. Wu and M. Shah and N. Lobo. A Virtual 3D Blackboard: 3D    Finger Tracking using a Single Camera, 2000, 536-543.-   [3] S. Malik, Real-time Hand Tracking and Finger Tracking for    Interaction. 2003.-   [4] C. Jennings. Robust Finger Tracking with Multiple Camera. In    Proc. of the International Workshop on Recognition, Analysis, and    Tracking of Faces and Gestures in Real-Time Systems, 1999, 152-160.-   [5] http://youtu.be/jerl Ms5dUJg Kinect Finger Tracking Video-   [6] http://touchco.com/Touchco-   [7] S. Jobs. 2008. Touch screen device, method, and graphical user    interface for determining commands by applying heuristics. U.S.    Patent Application No. 20080122796.-   [8] http://www.wacom.com/en/Products/Intuos.aspx Wacom Intuos 4-   [9] http://www.microsoft.com/surface/en/us/default.aspx Microsoft    Surface-   [10] http://www.merl.com/projects/DiamondTouch/Diamond Touch-   [11] http://smarttech.com/SMART Technologies-   [12] Y. Han. Low-cost multi-touch sensing through frustrated total    internal reflection. Proceedings of the 18th Annual ACM Symposium on    User Interface Software and Technology, pages 115-118,2005.-   [13] I. Rosenberg and K. Perlin. The unmousepad: an interpolating    multi-touch force-sensing input pad. In SIGGRAPH '09: ACM SIGGRAPH    2009 papers, pages 1-9, New York, N.Y., USA, 2009. ACM.-   [14] http://www.futuresource-consulting.com/Future Consulting-   [15] Ming-Ching Chang and Benjamin B. Kimia, Regularizing 3D Medial    Axis Using Medial Scaffold Transforms, Computer Vision and Pattern    Recognition, 2008. CVPR 2008. IEEE Conference.-   [16] Xia H. and Tucker P. G. (2009), Distance solutions for medial    axis transform, Proceedings of 18th International Meshing    Roundtable, Salt Lake City, Utah, Oct. 25-28, 2009, pp. 247-260.-   [17] http://www.umiacs.umd.edu/˜ramani/cmsc828d/lecture9.pdf, Ramani    Duraiswami, Lecture Notes on Camera Calibration, cmsc828d.-   [18] Multiple View Geometry in Computer Vision, R. Hartley and A.    Zisserman, Cambridge University Press, 2000, pp. 138-183.-   [19] Three-Dimensional Computer Vision: A Geometric Approach, O.    Faugeras, MIT Press, 1996, pp. 33-68.-   [20] A Versatile Camera Calibration Technique for 3D Machine    Vision”, R. Y. Tsai, IEEE J. Robotics & Automation, RA-3, No. 4,    August 1987, pp. 323-344.-   [21] Jean-Yves Bouguet, Camera Calibration Toolbox for Matlab,    http://www.vision.caltech.edu/bouguetj/calib doc.-   [22] Wayne Carl Westerman, John G. Elias, Maintaining activity after    contact liftoff or touchdown, U.S. Pat. No. 7,782,307.-   [23] Arther R. Pope, Model-Based Object Recognition—A Survey of    Recent Research, Technical Report 94-04 January 1994.-   [24] Farshid Arman, Model-based object recognition in dense-range    images—a review, Journal ACM Computing Surveys (CSUR) Surveys Volume    25 Issue 1, March 1993.-   [25] G. Qian, J. Zhang, and A. Kidane, “People Identification Using    Gait via Floor Pressure Analysis”, IEEE Sensors Journal, Vol. 10,    No. 9, pp. 1447-1460, September 2010, doi:    10.1109/JSEN.2010.2045158.-   [26] Augsten, T., Kaefer, K., Meusel, R., Fetzer, C., Kanitz, D.,    Stoff., T., Becker, T., Holz, C. and Baudisch, P. Multitoe:    High-Precision Interaction with Back-Projected Floors Based on    High-Resolution Multi-Touch Input. Proceedings of UIST 2010, New    York, N.Y., Oct. 3-6, 2010, pp. 209-218.-   [27] Cullen Jennings, Robust Finger Tracking with Multiple Cameras,    Proceedings of the International Workshop on Recognition, Analysis,    and Tracking of Faces and Gestures in Real-Time Systems, 1999.-   [28] Jacob O. Wobbrock, Meredith Ringel Morris, Andrew D. Wilson,    User-Defined Gestures for Surface Computing, Proceeding CHI '09    Proceedings of the 27th international conference on Human factors in    computing systems.

Although the invention has been described in detail in the foregoingembodiments for the purpose of illustration, it is to be understood thatsuch detail is solely for that purpose and that variations can be madetherein by those skilled in the art without departing from the spiritand scope of the invention except as it may be described by thefollowing claims.

1. An apparatus for inputting information into a computer comprising: a3D sensor that senses 3D information and produces a 3D output; a 2Dsensor that senses 2D information and produces a 2D output; and aprocessing unit which receives the 2D and 3D output and produces acombined output that is a function of the 2D and 3D output.
 2. Theapparatus of claim 1 wherein objects are identified and tracked in 3Dand 2D by the 3D and 2D sensors.
 3. The apparatus of claim 2 whereinfingers, hands, feet, people, pens or other objects are identified andtracked in 3D and 2D.
 4. The apparatus of claim 3 including a memory andwherein the identity of each object is maintained over time.
 5. Theapparatus of claim 4 wherein the identity of objects from the 3D sensorare paired with objects from the 2D sensor by the processing unit. 6.The apparatus of claim 5 wherein the 2D sensor has a surface and the 2Dsensor senses contact on the surface.
 7. The apparatus of claim 6wherein the 2D sensor senses imposed force on the surface.
 8. Theapparatus of claim 7 wherein the 2D sensor includes a pressure imagingsensor.
 9. The apparatus of claim 8 wherein the 3D sensor includes arange imaging camera.
 10. The apparatus of claim 8 wherein the 3D sensorincludes an IR depth camera.
 11. The apparatus of claim 8 wherein the 3Dsensor includes an RGB camera.
 12. A method for inputting informationinto a computer comprising: producing a 3d output with a 3d sensor thatsenses 3d information; producing a 2d output with a 2d sensor thatsenses 2d information; and producing a combined output with theprocessing unit that is a function of the 2d and 3d output.
 13. Themethod of claim 12 including the step of identifying and trackingobjects in 3D and 2D by the 3D and 2D sensors.
 14. The method of claim13 including the step of identifying and tracking fingers, hands, feet,people, pens or other objects in 3D and 2D.
 15. The method of claim 14including the step of maintaining in a memory the identity of eachobject over time.
 16. The method of claim 15 including the step ofpairing with the processing unit the identity of objects from the 3Dsensor with objects from the 2D sensor.
 17. The method of claim 16including the step of the 2D sensor senses contact on its surface. 18.The method of claim 17 including the step of the 2D sensor sensesimposed force on its surface.
 19. The method of claim 18 wherein the 2Dsensor includes a pressure imaging sensor.
 20. The method of claim 18wherein the 3D sensor includes a range imaging camera.